Reduce Testing Effort Research Articles

Distributed Petri nets for model-driven verifiable robotic applications in ROS

Verifying industrial robotic systems is a complex task because those systems are distributed and solely defined by their implementation instead of models of the system to be verified. Some technologies mitigate parts of this problem, e.g., robotic middleware such as the Robotic Operating System (ROS) or concrete solutions such as automata-based specification of robot behavior. However, they all lack the required modeling depth to describe the structure, behavior, and communication of the system. We introduce an improved version of our previous model-driven approach based on Petri nets, integrating these three aspects of ROS-based systems. Using a formal modeling language enables verification of the described system and the generation of complete system parts in the form of ROS nodes. This reduces testing effort because the specification of component workflows and interfaces remains formally proven, while only changed implementations have to be revalidated. We extended our previous approach with novel model transformations, which considerably improved our approach’s performance and memory requirements. We evaluate our approach in a case study involving multiple industrial robotic arms and show that the structure of and communication between ROS nodes can be described and verified.

Reduction of testing effort for fatigue tests: Application of Bayesian optimal experimental design

AbstractS‐N curve parameter determination is a time‐ and cost‐intensive procedure. A standardized method for simultaneously determining all S‐N curve parameters with minimum testing effort is still missing. The Bayesian optimal experimental design (BOED) approach can reduce testing effort and accelerates uncertainty reduction during fatigue testing for S‐N curve parameters. The concept is applicable to all S‐N curve models and is exemplary illustrated for a bilinear S‐N curve model. We demonstrate the fatigue testing workflow for the bilinear S‐N curve in detail while discussing steps and challenges when generalizing to other S‐N curve models. Applying the BOED to the bilinear S‐N curve models, minor errors and uncertainties for all S‐N curve parameters are obtained after only 10 experiments for data scatter values below 1.1. For such, the relative error in fatigue limit estimation was less than 1% after five tests. When S‐N data scatter higher than 1.2 is concerned, 17 tests were required for robust analysis. The BOED methodology should be applied to other S‐N curve models in the future. The high computational effort and the approximation of the posterior distribution with a normal distribution are the limitations of the presented BOED approach.

Machine learning-based test selection for simulation-based testing of self-driving cars software

Simulation platforms facilitate the development of emerging Cyber-Physical Systems (CPS) like self-driving cars (SDC) because they are more efficient and less dangerous than field operational test cases. Despite this, thoroughly testing SDCs in simulated environments remains challenging because SDCs must be tested in a sheer amount of long-running test cases. Past results on software testing optimization have shown that not all the test cases contribute equally to establishing confidence in test subjects’ quality and reliability, and the execution of “safe and uninformative” test cases can be skipped to reduce testing effort. However, this problem is only partially addressed in the context of SDC simulation platforms. In this paper, we investigate test selection strategies to increase the cost-effectiveness of simulation-based testing in the context of SDCs. We propose an approach called SDC-Scissor (SDC coS t-effeC tI ve teS t S electOR) that leverages Machine Learning (ML) strategies to identify and skip test cases that are unlikely to detect faults in SDCs before executing them. Our evaluation shows that SDC-Scissor outperforms the baselines. With the Logistic model, we achieve an accuracy of 70%, a precision of 65%, and a recall of 80% in selecting tests leading to a fault and improved testing cost-effectiveness. Specifically, SDC-Scissor avoided the execution of 50% of unnecessary tests as well as outperformed two baseline strategies. Complementary to existing work, we also integrated SDC-Scissor into the context of an industrial organization in the automotive domain to demonstrate how it can be used in industrial settings.

A Graph-Based Metadata Model for DevOps in Simulation-Driven Development and Generation of DCP Configurations

With the goal of improving the quality of model-based development and to reduce testing effort, DevOps practices have gained more and more importance. However, most system engineers are not DevOps specialists, and there are a lot of manual steps involved when writing build pipelines and configurations of simulations. For this purpose, an abstract graph-based metadata model is proposed. This allows the autogeneration of scenario descriptions for simulations and code for the build server where the simulation environment is set up and executed. This is demonstrated by applying this process to the DCP standard. In this paper, we will discuss three simple use cases which are motivated by practical problems that arise in complex development environments and how the proposed solutions can be used to tackle them. Detailed descriptions and implementations of the use cases show how the proposed methods can be applied in practice and help solve the described problems. Furthermore, a Python implementation of a DCP master and a simple FMI-to-DCP wrapper are presented in this work.

General influence of the environmental temperature on the matrix strength under tensile and compressive loading - A comprehensive study on high performance matrices

This comprehensive experimental study investigates the influence of operating temperatures ranging from −50 °C to 256 °C on the mechanical properties of various high-performance thermosets regularly used as matrix polymers for composites. The resins are either crucial for specific industry sectors or challenging to obtain because they are part of commercial prepreg systems. The research focuses on the temperature-dependent tensile and compressive strength of those materials and provides a wide database of mechanical properties. This paper summarizes the mechanical properties on the one hand in terms of a strength versus temperature scale, which allows an easy comparison for engineering applications. And on the other hand, it demonstrates that the thermo-mechanical properties are mainly dependent on the polymer's glass transition temperature. More precisely, three approaches to estimate the temperature dependence of the strength of thermosets with reduced testing effort are applied and presented. While one is based on the assumption that strength depends linearly on the environmental temperature, the others focus on the distance or the ratio of the environmental temperature to the glass transition temperature. This data set and the resulting correlations provide the community with valuable material data and allow to forecast the thermo-mechanical behavior of most thermosets in the future with significantly reduced effort.

A New Learning to Rank Approach for Software Defect Prediction

Software defect prediction is one of the most active research fields in software development. The outcome of defect prediction models provides a list of the most likely defect-prone modules that need a huge effort from quality assurance teams. It can also help project managers to effectively allocate limited resources to validating software products and invest more effort in defect-prone modules. As the size of software projects grows, error prediction models can play an important role in assisting developers and shortening the time it takes to create more reliable software products by ranking software modules based on their defects. Therefore, there is need a learning-to-rank approach that can prioritize and rank defective modules to reduce testing effort, cost, and time. In this paper, a new learning to rank approach was developed to help the QA team rank the most defect-prone modules using different regression models. The proposed approach was evaluated on a set of standardized datasets using well-known evaluation measures such as Fault-Percentile Average (FPA), Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and the Cumulative Lift Chart (CLC). Also, our proposed approach was compared with some other regression models that are used for software defect prediction, such as Random Forest (RF), Logistic Regression (LR), Support Vector Regression (SVR), Zero Inflated Regression (ZIR), Zero Inflated Poisson (ZIP), and Negative Polynomial Regression (NPR). Based on the results, the measurement criteria were different than each other as there was a gap in the accuracy obtained for defects prediction due to the nature of the random data, and thus was higher for RF and SVR, as well as FPA achieved better results than MAE and RMSE in this research paper.

Material defects detection based on in-process measurements in milling of Ti6246 alloy

An increasing demand for products and services in the aviation industry is attended by a rising demand for aircraft engines. This requires an increased productivity combined with highest quality and safety standards. In this paper, a new approach of material defect detection is presented which is based on force and acoustic-emission measurements during machining processes. Representative synthetic defects in titanium alloy Ti6246 were investigated. In order to use scale effects, the experiments were conducted in micro-milling processes. This approach could be used for an automated, non-destructive evaluation of the manufactured components in the future and thus for reducing testing effort and related auxiliary times.

Scented since the beginning: On the diffuseness of test smells in automatically generated test code

Software testing represents a key software engineering practice to ensure source code quality and reliability. To support developers in this activity and reduce testing effort, several automated unit test generation tools have been proposed. Most of these approaches have the main goal of covering as more branches as possible. While these approaches have good performance, little is still known on the maintainability of the test code they produce, i.e.,whether the generated tests have a good code quality and if they do not possibly introduce issues threatening their effectiveness. To bridge this gap, in this paper we study to what extent existing automated test case generation tools produce potentially problematic test code. We consider seven test smells, i.e.,suboptimal design choices applied by programmers during the development of test cases, as measure of code quality of the generated tests, and evaluate their diffuseness in the unit test classes automatically generated by three state-of-the-art tools such as Randoop, JTExpert, and Evosuite. Moreover, we investigate whether there are characteristics of test and production code influencing the generation of smelly tests. Our study shows that all the considered tools tend to generate a high quantity of two specific test smell types, i.e.,Assertion Roulette and Eager Test, which are those that previous studies showed to negatively impact the reliability of production code. We also discover that test size is correlated with the generation of smelly tests. Based on our findings, we argue that more effective automated generation algorithms that explicitly take into account test code quality should be further investigated and devised.

Partially safe evolution of software product lines

Software Product Lines allow the automatic generation of related products built with reusable artefacts. In this context, developers may need to perform changes and check whether products are affected. A strategy to perform such analysis is verifying behaviour preservation through the use of formal theories. The product line refinement notion requires behaviour preservation for all existing products. Nevertheless, in evolution scenarios like bug fixes, some products intentionally have their behaviour changed. To support developers in these and other unsafe scenarios, we define a theory of partial product line refinement that helps to precisely understand which products are affected by a change. This provides a kind of impact analysis that could, for example, reduce test effort, since only affected products need to be tested. We provide properties such as compositionality, which deals with changes to a specific product line element, and general properties to support developers when safe and partially safe scenarios are combined. We also define a set of transformation templates, which are classified according to their compatibility to specific types of product lines. To evaluate our work, we analyse two product lines: Linux and Soletta, to discover if our templates could be helpful in evolving these systems.

Coverage-based Approach for Model-based Testing in Software Product Line

Rapid Quality assurance is an important element in software testing in order to produce high quality products in Software Product Line (SPL). One of the testing techniques that can enhance product quality is Model-Based Testing (MBT). Due to MBT effectiveness in terms of reuse and potential to be adapted, this technique has become an efficient approach that is capable to handle SPL requirements. In this paper, the authors present an approach to manage variability and requirements by using Feature Model (FM) and MBT. This paper focuses on modelling the integration towards enhancing product quality and reducing testing effort. Further, the authors considered coverage criteria, including pairwise coverage, all-state coverage, and all-transition coverage, in order to improve the quality of products. For modelling purposes, the authors constructed a mapping model based on variability in FM and behaviour from statecharts. The proposed approach was validated using mobile phone SPL case study.

Similarity-based prioritization of test case automation

The importance of efficient software testing procedures is driven by an ever increasing system complexity as well as global competition. In the particular case of manual test cases at the system integration level, where thousands of test cases may be executed before release, time must be well spent in order to test the system as completely and as efficiently as possible. Automating a subset of the manual test cases, i.e, translating the manual instructions to automatically executable code, is one way of decreasing the test effort. It is further common that test cases exhibit similarities, which can be exploited through reuse when automating a test suite. In this paper, we investigate the potential for reducing test effort by ordering the test cases before such automation, given that we can reuse already automated parts of test cases. In our analysis, we investigate several approaches for prioritization in a case study at a large Swedish vehicular manufacturer. The study analyzes the effects with respect to test effort, on four projects with a total of 3919 integration test cases constituting 35,180 test steps, written in natural language. The results show that for the four projects considered, the difference in expected manual effort between the best and the worst order found is on average 12 percentage points. The results also show that our proposed prioritization method is nearly as good as more resource demanding meta-heuristic approaches at a fraction of the computational time. Based on our results, we conclude that the order of automation is important when the set of test cases contain similar steps (instructions) that cannot be removed, but are possible to reuse. More precisely, the order is important with respect to how quickly the manual test execution effort decreases for a set of test cases that are being automated.

State Transition Testing Approach for Ad hoc Networks using Ant Colony Optimization

Nowadays, telecommunication software organizations are challenged to provide high-quality software to customers within their estimated time and budget in order to stay competitive within the market. Because quality is a defining aspect of the product, it is essential for a project manager to stay alert throughout the project lifecycle. Quality has a direct bearing on customer satisfaction, and if a company produces high-quality products, satisfied customers will rank it highly in customer satisfaction surveys. Additionally, dissatisfied customers are more vocal in their criticisms. Therefore, testing is an important step to produce more reliable systems. In this paper we address two important aspects of software testing for ad hoc network protocols. The first one is by integrating a high-level testing approach based on state transition on top of a network simulator in order to fill a perceived gap in existing network simulators. The second one is reducing testing effort by eliminating redundant test cases, in order to effectively improve the result accuracy of existing network simulators. In this paper, we implemented an automated state transition testing approach for wireless network routing protocols, using an improved Ant Colony Optimization (ACO) algorithm. The expected result is to provide maximum coverage in terms of states and transitions.

Combined notch and size effect modeling in a local probabilistic approach for LCF

In recent years a local probabilistic model for low cycle fatigue (LCF) based on the statistical size effect has been developed and applied on engineering components. Here, the notch support extension based on the stress gradient effect is described in detail, as well as an FEA-based parameter calibration. An FEA is necessary to simulate non-homogeneous stress fields in non-smooth specimens which exhibit gradients and determine size effects. The hazard density approach and the surface integration over the FEA stress lead to geometry-independent model parameters. Three different materials (superalloys IN-939, Rene80, steel 26NiCrMoV14-5) and three different geometry types (smooth, notch, cooling hole specimens) are considered for a more comprehensive validation of the probabilistic LCF model and to demonstrate its wide application range. At the same time, a reduced testing effort is needed compared to deterministic model predictions with notch support.

Experimental prediction of sheet metal formability of AW-5754 for non-linear strain paths by using a cruciform specimen and a blank holder with adjustable draw beads on a sheet metal testing machine

The main objective to guarantee a high efficiency in the press shop is to produce sheet metal parts without failure. The feasibility of sheet metal parts is nowadays ensured during the development process by a comparison of the occurring strains in the simulation with the Forming Limit Diagram (FLD). The principle of the experimental procedure to determine the FLD is standardized in ISO 12004–2 [1]. This procedure is only valid with high accuracy for proportional unbroken strain paths. However, in most industrial forming operations non-linear strain paths occur. To resolve this problem, a phenomenological approach was introduced by Volk [2], the so-called Generalized Forming Limit Concept (GFLC). Localized necking and the remaining formability for any arbitrary non-linear strain path can be predicted with the GFLC. Furthermore, experimental investigation of multi-linear strain paths appears highly complex in practice and involves a range of testing equipment, e.g. different specimens, testing machines and tools. In this paper an alternative method is introduced by using a cruciform specimen and a draw bead tool on a sheet metal testing machine. The different draw bead heights allow the creation of arbitrary strain states, which can be changed at different height of the punch. Conventionally cruciform specimens are used to determine the yield loci in the first quadrant of the stress space at low strain values. To enable a cruciform specimen for the evaluation of strain limits comparable to the conventional Nakajima test, an optimization of the geometry regarding the maximum achievable strains in the specimen center takes place. The developed specimen and tool allow testing of materials under multi-axial strain states with a reduced testing effort.

Memory mutation testing

ContextThree decades of mutation testing development have given software testers a rich set of mutation operators, yet relatively few operators can target memory faults (as we demonstrate in this paper).ObjectiveTo address this shortcoming, we introduce Memory Mutation Testing, proposing 9 Memory Mutation Operators each of which targets common forms of memory fault. We compare Memory Mutation Operators with traditional Mutation Operators, while handling equivalent and duplicate mutants.MethodWe extend our previous workshop paper, which introduced Memory Mutation Testing, with a more extensive and precise analysis of 18 open source programs, including 2 large real-world programs, all of which come with well-designed unit test suites. Specifically, our empirical study makes use of recent results on Trivial Compiler Equivalence (TCE) to identify both equivalent and duplicate mutants. Though the literature on mutation testing has previously deployed various techniques to cater for equivalent mutants, no previous study has catered for duplicate mutants.ResultsCatering for such extraneous mutants improves the precision with which claims about mutation scores can be interpreted. We also report the results of a new empirical study that compares Memory Mutation Testing with traditional Mutation Testing, providing evidence to support the claim that traditional mutation testing inadequately captures memory faults; 2% of the memory mutants are TCE-duplicates of traditional mutants and average test suite effectiveness drops by 44% when the target shifts from traditional mutants to memory mutants.ConclusionsIntroducing Memory Mutation Operators will cost only a small portion of the overall testing effort, yet generate higher quality mutants compared with traditional operators. Moreover, TCE technique does not only help with reducing testing effort, but also improves the precision of assessment on test quality, therefore should be considered in other Mutation Testing studies.

Characterizing Permanent Deformation of Silty Sand Subgrades by Using a Model Based on Multistage Repeated-Load Triaxial Testing

A simple model for the prediction of permanent deformation of unbound pavement material was used to characterize the accumulation of permanent deformation of unsaturated silty subgrade soils from multistage repeated-load triaxial (RLT) tests. The model incorporated the time-hardening concept; data obtained from RLT tests with cyclic loading of variable magnitudes, conducted on a single specimen, could be used to calibrate the material model parameters. This feature of the model allowed for characterizing the accumulation of permanent deformation of the material under a wide range of stress conditions. The tests were carried out on two silty sand subgrades and at different moisture contents with pore suction measurements (matric suction) incorporated into the stress state calculations used in the modeling. The shakedown ranges for the permanent deformations for each stress path were calculated, and the material parameters of the predictive models were optimized with the RLT test data. In general, the models performed satisfactorily in capturing the permanent deformation behavior of the selected subgrade materials. The model includes fewer parameters as compared with some existing permanent deformation models and requires reduced testing effort because it is based on results from a multistage RLT test with a single specimen. It can be concluded that the model and the approach have great potential for characterizing and modeling the permanent deformation of subgrade soils.

Refinement типы для языка Jolie

Jolie is the first language for microservices and it is currently dynamically type checked. This paper considers the opportunity to integrate dynamic and static type checking with the introduction of refinement types, verified via an SMT solver. The integration of the two aspects allows a scenario where the static verification of internal services and the dynamic verification of (potentially malicious) external services cooperate in order to reduce testing effort and enhance security. Refinement types are well-known technique for numeric, array and algebraic data types. They rely on corresponding SMT-theories. Recently SMT solvers got support for a theory of strings and regular expressions. In the paper, we describe possible application of the theory to string refinement types. We use Jolie programming language to illustrate feasibility and usefulness of such extension. First, because Jolie already has syntax extension to support string refinements. We build on top of that extension to provide static type checking. Second, because in the realm of microservices the need for improved checking of string data is much higher as most of external communication goes through text-based protocols. We present simplified but real-world example from the domain of web-development. We intentionally introduce a bug in the example demonstrating how easily it can slip a conventional type system. Proposed solution is feasible, as it do not accept program with the bug. Complete solution will need enhancements in precision and error reporting.

Guest editors' introduction

Guest editors' introduction

When to automate software testing? A decision‐support approach based on process simulation

AbstractSoftware test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, because test automation usually requires substantial upfront investments, automation is not always more cost‐effective than manual testing. To support decision‐makers in finding the optimal degree of test automation in a given project, we recently proposed a process simulation model using the System Dynamics modeling technique and used the simulation model in the context of a case study with a software company in Calgary, Canada. With the help of the simulation model, we were able to evaluate the performance of test processes with varying degrees of automation of test activities and help testers choose the most optimal cases. The goal of the earlier study was to investigate how the simulation model can help decision‐makers decide whether and to what degree the company should automate their test processes. In this article, we present further details of the System Dynamics model, its usage scenarios and examples of simulation experiments independent from a specific company context. Copyright © 2015 John Wiley & Sons, Ltd.

Realistic Fe simulation of foldcore sandwich structures

Foldcore is an origami-like structural sandwich core which is manufactured by folding a planar base material into a three dimensional structure. The manufacturing technology is open to a variety of base materials and also a range of unit cell geometries is feasible. This results in a wide spectrum of mechanical properties which can be achieved by foldcores. Mechanical characterisation of foldcores by testing is therefore costly. So FE simulation is required to reduce testing effort. A modelling method for foldcores which allows realistic simulation is developed and presented within this paper. The modelling method is validated by comparison with experiments. The test cases are compression tests and shear tests. Stiffness and strength is evaluated from experiment and simulation. Comparison shows very good correlation. The presented method is able to simulate the mechanical properties of foldcores and is therefore a valuable tool for further product development.