Source Test Cases Research Articles

Improving Early Fault Detection in Machine Learning Systems Using Data Diversity-Driven Metamorphic Relation Prioritization

Metamorphic testing is a valuable approach to verifying machine learning programs where traditional oracles are unavailable or difficult to apply. This paper proposes a technique to prioritize metamorphic relations (MRs) in metamorphic testing for machine learning and deep learning systems, aiming to enhance early fault detection. We introduce five metrics based on diversity in source and follow-up test cases to prioritize MRs. The effectiveness of our proposed prioritization methods is evaluated on three machine learning and one deep learning algorithm implementation. We compare our approach against random-based, fault-based, and neuron activation coverage-based MR ordering. The results show that our data diversity-based prioritization performs comparably to fault-based prioritization, reducing fault detection time by up to 62% compared to random MR execution. Our proposed metrics outperformed neuron activation coverage-based prioritization, providing 5–550% higher fault detection effectiveness. Overall, our approach to prioritizing metamorphic relations leads to increased fault detection effectiveness and reduced average fault detection time. This improvement in efficiency can result in significant time and cost savings when applying metamorphic testing to machine learning and deep learning systems.

SFIDMT-ART: A metamorphic group generation method based on Adaptive Random Testing applied to source and follow-up input domains

Context:The performance of metamorphic testing relates strongly to the quality of test cases. However, most related research has only focused on source test cases, ignoring follow-up test cases to some extent. In this paper, we identify a potential problem that may be encountered with existing metamorphic group generation algorithms. We then propose a possible solution to address this problem. Based on this solution, we design a new algorithm for generating effective source and follow-up test cases. Objective:To improve the performance (test effectiveness and efficiency) of metamorphic testing. Methods:We introduce the concept of the input-domain difference problem, which is likely to affect the performance of metamorphic group generation algorithms. We propose a new test-case distribution criterion for metamorphic testing to address this problem. Based on our proposed criterion, we further present a new metamorphic group generation algorithm, from a black-box perspective, with new distance metrics to facilitate this algorithm. Results:Our algorithm performs significantly better than existing algorithms, in terms of test effectiveness, efficiency and test-case diversity. Conclusions:Through experiments, we find that the input-domain difference problem is likely to affect the performance of metamorphic group generation algorithms. The experimental results demonstrate that our algorithm can achieve good test efficiency, effectiveness, and test-case diversity.

Automated code-based test case reuse for software product line testing

ContextA software product line (SPL) grows in size as a new product is developed. A new product in an SPL should be tested extensively for quality assurance. For the efficient testing, previous studies suggested reusing the existing test cases of a product family. However, either their methods were not efficient because interventions from human experts, specifications, architecture and/or traceabilities for test cases were required. ObjectiveTo address these limitations, we propose an Automated Code-based Test case reuse for SPLs (ActSPL). ActSPL automatically identifies reusable test cases for new products of a product family using source code and test cases. MethodActSPL automatically constructs a hash-based traceability links between test cases and source code of a product family. Using the traceability links, ActSPL selects reusable test cases for a given new product from existing test cases of the product family. ResultsWe evaluated ActSPL in terms of the effectiveness and cost reduction of reusing test cases with five open-source SPLs. The evaluation results showed that ActSPL, on average, achieved 100 % precision and 62 % recall. In addition, ActSPL, on average, saved 47.5 % of time required for testing a new product from scratch. ConclusionOur study shows the feasibility of ActSPL reusing SPL test cases based on source code and test cases. Our results can be a basis for successive studies for automated code-based SPL testing.

An Automated Framework for Detecting Change in the Source Code and Test Case Change Recommendation

Improvements and acceleration in software development have contributed towards high-quality services in all domains and all fields of industry, causing increasing demands for high-quality software developments. The industry is adopting human resources with high skills, advanced methodologies, and technologies to match the high-quality software development demands to accelerate the development life cycle. In the software development life cycle, one of the biggest challenges is the change management between the version of the source codes. Various reasons, such as changing the requirements or adapting available updates or technological upgrades, can cause the source code's version. The change management affects the correctness of the software service's release and the number of test cases. It is often observed that the development life cycle is delayed due to a lack of proper version control and due to repetitive testing iterations. Hence the demand for better version control-driven test case reduction methods cannot be ignored. The parallel research attempts propose several version control mechanisms. Nevertheless, most version controls are criticized for not contributing toward the test case generation of reduction. Henceforth, this work proposes a novel probabilistic rule-based test case reduction method to simplify the software development's testing and version control mechanism. Software developers highly adopt the refactoring process for making efficient changes such as code structure and functionality or applying changes in the requirements. This work demonstrates very high accuracy for change detection and management. This results in higher accuracy for test case reductions. The outcome of this work is to reduce the development time for the software to make the software development industry a better and more efficient world.

Feedback-Directed Metamorphic Testing

Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a set of metamorphic relations among multiple software inputs and their expected outputs, metamorphic testing not only provides a test case generation strategy by constructing new (or follow-up) test cases from some original (or source) test cases, but also a test result verification mechanism through checking the relationship between the outputs of source and follow-up test cases. Many efforts have been made to further improve the cost-effectiveness of metamorphic testing from different perspectives. Some studies attempted to identify “good” metamorphic relations, while other studies were focused on applying effective test case generation strategies especially for source test cases. In this article, we propose improving the cost-effectiveness of metamorphic testing by leveraging the feedback information obtained in the test execution process. Consequently, we develop a new approach, namely feedback-directed metamorphic testing, which makes use of test execution information to dynamically adjust the selection of metamorphic relations and selection of source test cases. We conduct an empirical study to evaluate the proposed approach based on four laboratory programs, one GNU program, and one industry program. The empirical results show that feedback-directed metamorphic testing can use fewer test cases and take less time than the traditional metamorphic testing for detecting the same number of faults. It is clearly demonstrated that the use of feedback information about test execution does help enhance the cost-effectiveness of metamorphic testing. Our work provides a new perspective to improve the efficacy and applicability of metamorphic testing as well as many other software testing techniques.

Online Correction of Camera Poses for the Surround-view System: A Sparse Direct Approach

The surround-view module is an indispensable component of a modern advanced driving assistance system. By calibrating the intrinsics and extrinsics of the surround-view cameras accurately, a top-down surround-view can be generated from raw fisheye images. However, poses of these cameras sometimes may change. At present, how to correct poses of cameras in a surround-view system online without re-calibration is still an open issue. To settle this problem, we introduce the sparse direct framework and propose a novel optimization scheme of a cascade structure. This scheme is actually composed of two levels of optimization and two corresponding photometric error based models are proposed. The model for the first-level optimization is called the ground model, as its photometric errors are measured on the ground plane. For the second level of the optimization, it’s based on the so-called ground-camera model, in which photometric errors are computed on the imaging planes. With these models, the pose correction task is formulated as a nonlinear least-squares problem to minimize photometric errors in overlapping regions of adjacent bird’s-eye-view images. With a cascade structure of these two levels of optimization, an appropriate balance between the speed and the accuracy can be achieved. Experiments show that our method can effectively eliminate the misalignment caused by cameras’ moderate pose changes in the surround-view system. Source code and test cases are available online at https://cslinzhang.github.io/CamPoseCorrection/ .

MT-ART: A Test Case Generation Method Based on Adaptive Random Testing and Metamorphic Relation

Most of metamorphic testing (MT) research works focused on the generation and application of metamorphic relations (MRs). There is no clear conclusion about the relationship between test case generation methods and performance of MT. In this article, we introduce a novel method based on adaptive random testing (ART) and MR for MT test case generation. It proposes a family of algorithms for MT test cases generation, named as MT based ART (MT-ART). Three distances are measured to generate the next MT test case. In order to verify the performance of this method, series of experiments on four programs with different numbers of inputs are introduced. The results show that MT-ART performs better than other ART algorithms not only in test effectiveness, but also in test efficiency and test coverage. Based on this article, the following conclusions can be drawn: first, considering the effectiveness of MRs and test cases in MT may lead to better results. In this way, most of the existing research can be improved by this method. This is the most important contribute of our research. Second, not only the source test cases, but also the follow-up test cases can improve the performance of MT. Therefore, they should be considered together during the process of the next test case generation. Third, the average distance performs better than the max distance and the minimum distance in metamorphic test case selection.

Path-directed source test case generation and prioritization in metamorphic testing

Metamorphic testing is a technique that makes use of some necessary properties of the software under test, termed as metamorphic relations, to construct new test cases, namely follow-up test cases, based on some existing test cases, namely source test cases. Due to the ability of verifying testing results without the need of test oracles, it has been widely used in many application domains and detected lots of real-life faults. Numerous investigations have been conducted to further improve the effectiveness of metamorphic testing, most of which were focused on the identification and selection of “good” metamorphic relations. Recently, a few studies emerged on the research direction of how to generate and select source test cases that are effective in fault detection. In this paper, we propose a novel approach to generating source test cases based on their associated path constraints, which are obtained through symbolic execution. The path distance among test cases is leveraged to guide the prioritization of source test cases, which further improve the efficiency. A tool has been developed to automate the proposed approach as much as possible. Empirical studies have also been conducted to evaluate the fault-detection effectiveness of the approach. The results show that this approach enhances both the performance and automation of metamorphic testing. It also highlights interesting research directions for further improving metamorphic testing.

Revisiting Test Impact Analysis in Continuous Testing From the Perspective of Code Dependencies

In continuous testing, developers execute automated test cases once or even several times per day to ensure the quality of the integrated code. Although continuous testing helps ensure the quality of the code and reduces maintenance effort, it also significantly increases test execution overhead. In this paper, we empirically evaluate the effectiveness of test impact analysis from the perspective of code dependencies in the continuous testing setting. We first applied test impact analysis to one year of software development history in 11 large-scale open-source systems. We found that even though the number of changed files is small in daily commits (median ranges from 3 to 28 files), around 50 percent or more of the test cases are still impacted and need to be executed. Motivated by our finding, we further studied the code dependencies between source code files and test cases, and among test cases. We found that 1) test cases often focus on testing the integrated behaviour of the systems and 15 percent of the test cases have dependencies with more than 20 source code files; 2) 18 percent of the test cases have dependencies with other test cases, and test case inheritance is the most common cause of test case dependencies; and 3) we documented four dependency-related test smells that we uncovered in our manual study. Our study provides the first step towards studying and understanding the effectiveness of test impact analysis in the continuous testing setting and provides insights on improving test design and execution.

Adaptive metamorphic testing with contextual bandits

Metamorphic Testing is a software testing paradigm which aims at using necessary properties of a system under test, called metamorphic relations, to either check its expected outputs, or to generate new test cases. Metamorphic Testing has been successful to test programs for which a full oracle is not available or to test programs for which there are uncertainties on expected outputs such as learning systems. In this article, we propose Adaptive Metamorphic Testing as a generalization of a simple yet powerful reinforcement learning technique, namely contextual bandits, to select one of the multiple metamorphic relations available for a program. By using contextual bandits, Adaptive Metamorphic Testing learns which metamorphic relations are likely to transform a source test case, such that it has higher chance to discover faults. We present experimental results over two major case studies in machine learning, namely image classification and object detection, and identify weaknesses and robustness boundaries. Adaptive Metamorphic Testing efficiently identifies weaknesses of the tested systems in context of the source test case.

An Automated Framework for Detecting Change in the Source Code and Test Case Change Recommendation

Improvements and acceleration in software development has contributed towards high quality services in all domains and all fields of industry causing increasing demands for high quality software developments. In order to match with the high-quality software development demands, the software development industry is adopting human resources with high skills, advanced methodologies and technologies for accelerating the development life cycle. In the software development life cycle, one of the biggest challenges is the change management between versions of the source codes. The versing of the source code can be caused by various reasons such as change in the requirements or adaptation of functional update or technological upgradations. The change management does not only affect the correctness of the release for the software service, rather also impact the number of test cases. It is often observed that, the development life cycle is delayed due to lack of proper version control and due to the improver version control, the repetitive testing iterations. Hence the demand for better version control driven test case reduction methods cannot be ignored. A number of version control mechanisms are proposed by the parallel research attempts. Nevertheless, most of the version controls are criticized for not contributing towards the test case generation of reduction. Henceforth, this work proposes a novel probabilistic refactoring detection and rule-based test case reduction method in order to simplify the testing and version control mechanism for the software development. The refactoring process is highly adopted by the software developers for making efficient changes such as code structure, functionality or apply change in the requirements. This work demonstrates a very high accuracy for change detection and management. This results into a higher accuracy for test case reductions. The final outcome of this work is to reduce the development time for the software for making the software development industry a better and efficient world.

Automated code-based test selection for software product line regression testing

Regression testing for software product lines (SPLs) is challenging and can be expensive because it must ensure that all the products of a product family are correct whenever changes are made. SPL regression testing can be made efficient through a test case selection method that selects only the test cases relevant to the changes. Some approaches for SPL test case selection have been proposed but either they were not efficient by requiring intervention from human experts or they cannot be used if requirements specifications, architecture and/or traceabilities for test cases are not available or partially eroded. To address these limitations, we propose an automated method of source code-based regression test selection for SPLs. Our method reduces the repetition of the selection procedure and minimizes the in-depth analysis effort for source code and test cases based on the commonality and variability of a product family. Evaluation results of our method using six product lines show that our method reduces the overall time to perform regression testing by 14.8% ∼ 49.1% on average compared to an approach of repetitively applying Ekstazi, which is the state-of-the-art regression test selection method for a single product, to each product of a product family.

An Airgun Array Source Model Accounting for High-Frequency Sound Emissions During Firing—Solutions to the IAMW Source Test Cases

JASCO's Airgun Array Source Model (AASM) is a combined deterministic and stochastic model that separately treats the low-frequency and high-frequency components of signals produced by airgun arrays. The low-frequency module is based on solving the equations of motion for interacting spherical bubbles. The high-frequency module is based on a stochastic model of the airgun spectrum, which has been derived from a principal component regression analysis of the experimental data. This stochastic model determines the frequency spectrum of an airgun waveform during the rapid onset of a pressure that occurs when air is released from the gun chamber. AASM combines the output of these two modules to predict the source waveform of an airgun array over a wide frequency range (0-25 kHz). AASM was among the source models included in benchmark comparisons presented at the International Airgun Modeling Workshop (IAMW), held in Dublin, Ireland, in 2016. Results from the workshop showed that different source models agreed reasonably well at low frequencies (<;200 Hz), but they diverged substantially at high frequencies (>1 kHz). To help better understand the reasons for a mismatch between source models, this paper presents solutions to the IAMW source test cases, calculated using AASM, as well as a detailed description of AASM's theoretical underpinnings.

Novel Open Source Python Neutrosophic Package

Neutrosophic sets has gained wide popularity and acceptance in both academia and industry. Different fields have successfully adopted and utilized neutrosophic sets. However, there is no open-source implementation that provides the basic neutrosophic concepts. Open-source is a global movement that enables developers to share their source-code, as researchers do share their research ideas and results. Presented Open-source python neutrosophic package is the first of its kind. It utilizes object oriented concepts. It is based on one of the most popular multiparadigm programming languages that is widely used in different academic and industry fields and activities; namely python. Presented package tends to dissolve the barriers and enable both researchers and developers to adopt neutrosophic sets and theory in research and applications. In this paper, the first Open-source, Object-Oriented, Python based Neutrosophic package is presented. This paper intensively scanned neutrosophic sets research attempting to reach the most widely accepted neutrosophic proofs, and then transforming them into source-code. Presented package implements most of the basic neutrosophic concepts. Presented neutrosophic package presents four different classes: Single Valued Neutrosophic Number, Single Valued Neutrosophic Sets, Interval Valued Neutrosophic Number, and Interval Valued Neutrosophic Sets. Presented package source code, test cases, usage examples, and updated documentation can be found online at https://www.github.com/helghareeb/neutrosophic. Presented neutrosophic package can be easily integrated into research and applications. This is an ongoing work and research, as neutrosophic theory is largely expanding, and there are lots of features to cover.

Faultprog: Testing the Accuracy of Binary-Level Software Fault Injection

Off-The-Shelf (OTS) software components are the cornerstone of modern systems, including safety-critical ones. However, the dependability of OTS components is uncertain due to the lack of source code, design artifacts and test cases, since only their binary code is supplied. Fault injection in components’ binary code is a solution to understand the risks posed by buggy OTS components. In this paper, we consider the problem of the accurate mutation of binary code for fault injection purposes. Fault injection emulates bugs in high-level programming constructs (assignments, expressions, function calls, ...) by mutating their translation in binary code. However, the semantic gap between the source code and its binary translation often leads to inaccurate mutations. We propose Faultprog , a systematic approach for testing the accuracy of binary mutation tools. Faultprog automatically generates synthetic programs using a stochastic grammar, and mutates both their binary code with the tool under test, and their source code as reference for comparisons. Moreover, we present a case study on a commercial binary mutation tool, where Faultprog was adopted to identify code patterns and compiler optimizations that affect its mutation accuracy.

MOCCASIN: converting MATLAB ODE models to SBML.

Summary: MATLAB is popular in biological research for creating and simulating models that use ordinary differential equations (ODEs). However, sharing or using these models outside of MATLAB is often problematic. A community standard such as Systems Biology Markup Language (SBML) can serve as a neutral exchange format, but translating models from MATLAB to SBML can be challenging—especially for legacy models not written with translation in mind. We developed MOCCASIN (Model ODE Converter for Creating Automated SBML INteroperability) to help. MOCCASIN can convert ODE-based MATLAB models of biochemical reaction networks into the SBML format.Availability and implementation: MOCCASIN is available under the terms of the LGPL 2.1 license (http://www.gnu.org/licenses/lgpl-2.1.html). Source code, binaries and test cases can be freely obtained from https://github.com/sbmlteam/moccasin.Contact: mhucka@caltech.eduSupplementary information: More information is available at https://github.com/sbmlteam/moccasin.

A user-oriented procedure for the commissioning and quality assurance testing of treatment planning system dosimetry in high-dose-rate brachytherapy

PurposeTo develop a user-oriented procedure for testing treatment planning system (TPS) dosimetry in high-dose-rate brachytherapy, with particular focus to TPSs using model-based dose calculation algorithms (MBDCAs). Methods and MaterialsIdentical plans were prepared for three computational models using two commercially available systems and the same 192Ir source. Reference dose distributions were obtained for each plan using the MCNP v.6.1 Monte Carlo (MC) simulation code with input files prepared via automatic parsing of plan information using a custom software tool. The same tool was used for the comparison of reference dose distributions with corresponding MBDCA exports. ResultsThe single source test case yielded differences due to the MBDCA spatial discretization settings. These affect points at relatively increased distance from the source, and they are abated in test cases with multiple source dwells. Differences beyond MC Type A uncertainty were also observed very close to the source(s), close to the test geometry boundaries, and within heterogeneities. Both MBDCAs studied were found equivalent to MC within 5 cm from the target volume for a clinical breast brachytherapy test case. These are in agreement with previous findings of MBDCA benchmarking in the literature. ConclusionsThe data and the tools presented in this work, that are freely available via the web, can serve as a benchmark for advanced clinical users developing their own tests, a complete commissioning procedure for new adopters of currently available TPSs using MBDCAs, a quality assurance testing tool for future updates of already installed TPSs, or as an admission prerequisite in multicentric clinical trials.

A quadtree-adaptive multigrid solver for the Serre–Green–Naghdi equations

The Serre–Green–Naghdi (SGN) equations, also known as the fully-nonlinear Boussinesq wave equations, accurately describe the behaviour of dispersive shoaling water waves. This article presents and validates a novel combination of methods for the numerical approximation of solutions to the SGN equations. The approach preserves the robustness of the original finite-volume Saint-Venant solver, in particular for the treatment of wetting/drying and equilibrium states. The linear system of coupled vector equations governing the dispersive SGN momentum sources is solved simply and efficiently using a generic multigrid solver. This approach generalises automatically to adaptive quadtree meshes. Adaptive mesh refinement is shown to provide orders-of-magnitude gains in speed and memory when applied to the dispersive propagation of waves during the Tohoku tsunami. The source code, test cases and examples are freely available.