Automated Regression Testing Research Articles

Test case selection and prioritization approach for automated regression testing using ontology and COSMIC measurement

Test case selection and prioritization approach for automated regression testing using ontology and COSMIC measurement

Automated regression test method for scientific computing libraries: Illustration with SPHinXsys

Automated regression test method for scientific computing libraries: Illustration with SPHinXsys

Pengujian Regresi Otomasi pada Aplikasi Mobile Satudikti Menggunakan Katalon Studio

Satudikti is a one-stop application from the many services available at the Directorate General of Higher Education, to make it easier for users to access various higher education services in one application. Before the application can be widely used, a testing step is indispensable. This is due to potential problems in higher education data that can have a fatal impact, considering that this data is closely related to state information. And if there is an error in the software, the Directorate General of Higher Education must restart the development process that can harm the state. Therefore, the purpose of this study is to test the readiness of the application when it is widely used and reduce the risk of errors to ensure that the application remains of sufficient quality for users, as well as analyze the effectiveness of the application and then whether automated regression testing using Katalon Studio has more value than manual testing. Testing in this study was carried out manually with the black-box testing method and automated testing using Katalon Studio as an automated testing tool. Testing using Katalon Studio found that the functionality, features, and access to the tested pages worked well according to existing requirements. The implementation of automation testing using Katalon Studio has also proven to be more efficient in reducing test time and providing detailed test results to detect errors with easy-to-understand reporting.

Estimation of convergence of spectral element method in CAE Fidesys based on exact solution of the Lame problem forelastoplastic materials using an automated regression testing system

Estimation of convergence of spectral element method in CAE Fidesys based on exact solution of the Lame problem forelastoplastic materials using an automated regression testing system

Автоматическое тестирование LLVM-программ со сложными входными структурами данных

Symbolic execution is a widely used approach for automatic regression test generation and bug and vulnerability finding. The main goal of this paper is to present a practical symbolic execution-based approach for LLVM programs with complex input data structures. The approach is based on the well-known idea of lazy initialization, which frees the user from providing constraints on input data structures manually. Thus, it provides us with a fully automatic symbolic execution of even complex program. Two lazy initialization improvements are proposed for segmented memory models: one based on timestamps and one based on type information. The approach is implemented in the KLEE symbolic virtual machine for the LLVM platform and tested on real C data structures — lists, binomial heaps, AVL trees, red-black trees, binary trees, and tries.

Testing Numbs Us to Our Loss of Intellectual Control

It seems that today, with all of our tests, we allow the code to grow woolly and complicated in ways that, in the past, with no automated regression tests, we never could have tolerated. We were forced to keep the code simple because if we didn't, the complexity of the special cases wouldn't fit in our heads. This leads me to the following idea, which I will dig into in the rest of this article: Software teams need a healthy balance of both intellectual control, which comes from reasoning, and statistical control, which comes from testing. Complexity is the enemy of reasoning; efforts to maintain intellectual control tend to push complexity down. In my experience, many teams let their intellectual control atrophy and then compensate with more testing. This approach works for a while, but without intellectual control to keep complexity down, progress becomes slower and more difficult. Once lost, intellectual control is expensive to recover, so the teams find themselves in a local maximum they cannot escape.

Automated Regression Testing for Web Services

Web services represent class of applications developed through open internet standards. They help to develop dynamic applications and thus are used to deliver business components or functionalities on the web. Business functionalities as per changing market scenarios are frequently changing and so are these web services. It is therefore the need to properly ensure quality of these web services as they are essential to deliver functions which are required for organizations in their strategic business. To maintain desired level of quality, software maintainability of such applications is important. Regression Testing is required for proper maintaining any software and it becomes very costly in terms of cost and labor for constantly changing applications. Thus to reduce maintenance cost we need to reduce effort required in such testing. This paper presents an efficient regression testing strategy which is also automated to reduce the regression costs of web services. The strategy is implemented through an extensive framework supported by automation and also integrates data testing with regression. The proposed work is particularly useful for those applications which are delivered over web.

Automated Regression Test Case Generation for Web Application

Testing is one of the most important phases in the development of any product or software. Various types of software testing exist that have to be done to meet the need of the software. Regression testing is one of the crucial phases of testing where testing of a program is done for the original test build along with the modifications. In this article, various studies proposed by the authors have been analysed focusing on test cases generation and their approach toward web application. A detailed study was conducted on Regression Test Case Generation and its approaches toward web application. From our detailed study, we have found that very few approaches and methodologies have been found that provide the real tool for test case generation. There is a need of an automated regression testing tool to generate the regression test cases directly based on user requirements. These test cases have to be generated and implemented by the tool so that the reduction in the overall effort and cost can be achieved. From our study, we have also found that regression testing for web applications was not investigated much, but in today's scenario web applications are an integral part of our daily life and so that needs to be tested for regression testing.

Image-Based Approach to Determining Regression Test Results of Dynamic Web Applications

Much work has been done on automating regression testing for applications. But most of them focus on test execution. Little work has been done on automatically determining if a test case passes or fails. This decision is often made by comparing the results of executing test cases on a base version of the application and post-modification version of the application. If the two results match, the test case passes, otherwise fails. However, to the best of our knowledge, there is no regression testing method for automatically deciding pass/fail of dynamic Web applications which use JavaScript or CSS. We propose a method that automatically decides if a dynamic Web application passes a regression test case. The basic idea is to obtain a screenshot each time the GUI of the Web application (i.e. Web page) changes its state, and then compare each corresponding screenshot to see if they match. The evaluation results showed that the accuracy rate of our approach is high and our approach can be considered as fast enough for practical use.

Agile Acceptance Test-Driven Development of Clinical Decision Support Advisories: Feasibility of Using Open Source Software.

BackgroundMoving to electronic health records (EHRs) confers substantial benefits but risks unintended consequences. Modern EHRs consist of complex software code with extensive local configurability options, which can introduce defects. Defects in clinical decision support (CDS) tools are surprisingly common. Feasible approaches to prevent and detect defects in EHR configuration, including CDS tools, are needed. In complex software systems, use of test–driven development and automated regression testing promotes reliability. Test–driven development encourages modular, testable design and expanding regression test coverage. Automated regression test suites improve software quality, providing a “safety net” for future software modifications. Each automated acceptance test serves multiple purposes, as requirements (prior to build), acceptance testing (on completion of build), regression testing (once live), and “living” design documentation. Rapid-cycle development or “agile” methods are being successfully applied to CDS development. The agile practice of automated test–driven development is not widely adopted, perhaps because most EHR software code is vendor-developed. However, key CDS advisory configuration design decisions and rules stored in the EHR may prove amenable to automated testing as “executable requirements.”ObjectiveWe aimed to establish feasibility of acceptance test–driven development of clinical decision support advisories in a commonly used EHR, using an open source automated acceptance testing framework (FitNesse).MethodsAcceptance tests were initially constructed as spreadsheet tables to facilitate clinical review. Each table specified one aspect of the CDS advisory’s expected behavior. Table contents were then imported into a test suite in FitNesse, which queried the EHR database to automate testing. Tests and corresponding CDS configuration were migrated together from the development environment to production, with tests becoming part of the production regression test suite.ResultsWe used test–driven development to construct a new CDS tool advising Emergency Department nurses to perform a swallowing assessment prior to administering oral medication to a patient with suspected stroke. Test tables specified desired behavior for (1) applicable clinical settings, (2) triggering action, (3) rule logic, (4) user interface, and (5) system actions in response to user input. Automated test suite results for the “executable requirements” are shown prior to building the CDS alert, during build, and after successful build.ConclusionsAutomated acceptance test–driven development and continuous regression testing of CDS configuration in a commercial EHR proves feasible with open source software. Automated test–driven development offers one potential contribution to achieving high-reliability EHR configuration. Vetting acceptance tests with clinicians elicits their input on crucial configuration details early during initial CDS design and iteratively during rapid-cycle optimization.

Automated regression testing of BPMN 2.0 processes: a capture and replay framework for continuous delivery

Regression testing is a form of software quality assurance (QA) that involves comparing the behavior of a newer version of a software artifact to its earlier correct behavior, and signaling the QA engineer when deviations are detected. Given the large potential in automated generation and execution of regression test cases for business process models in the context of running systems, powerful tools are required to make this practically feasible, more specifically to limit the potential impact on production systems, and to reduce the manual effort required from QA engineers. In this paper, we present a regression testing automation framework that implements the capture & replay paradigm in the context of BPMN 2.0, a domain-specific language for modeling and executing business processes. The framework employs parallelization techniques and efficient communication patterns to reduce the performance overhead of capturing. Based on inputs from the QA engineer, it manipulates the BPMN2 model before executing tests for isolating the latter from external dependencies (e.g. human actors or expensive web services) and for avoiding undesired side-effects. Finally, it performs a regression detection algorithm and reports the results to the QA engineer. We have implemented our framework on top of a BPMN2-compliant execution engine, namely jBPM, and performed functional validations and evaluations of its performance and fault-tolerance. The results, indicating 3.9% average capturing performance overhead, demonstrate that the implemented framework can be the foundation of a practical regression testing tool for BPMN 2.0, and a key enabler for continuous delivery of business process-driven applications and services.

RLEImage: run-length encoded memory compression scheme for an itk::Image

This document describes a new class, itk::RLEImage, which uses run-length encoding to reduce the memory needed for storage of label maps. This class is accompanied by all the iterators to make it a dropin replacement for itk::Image. By changing the image typedef to itk::RLEImage, many ITK image processing algorithms build without modification and with minimal performance overhead. However, it is not possible if the user code uses GetBufferPointer() or otherwise assumes a linear pixel layout.This class is implemented to reduce the memory use of ITK-SNAP (www.itksnap.org), so ITKSNAP is the base for measuring the quantitative results.The class, accompanying iterator specializations, automated regression tests, and test data are all packaged as an ITK remote module https://github.com/KitwareMedical/ITKRLEImage.

ND morphological contour interpolation

This document describes a new class, itk::MorphologicalContourInterpolator, which implements a method proposed by Albu et al. in 2008. Interpolation is done by first determining correspondence between shapes on adjacent segmented slices by detecting overlaps, then aligning the corresponding shapes, generating transition sequence of one-pixel dilations and taking the median as result. Recursion is employed if the original segmented slices are separated by more than one empty slice.This class is n-dimensional, and supports inputs of 3 or more dimensions. `Slices’ are n-1-dimensional, and can be both automatically detected and manually set. The class is efficient in both memory used and execution time. It requires little memory in addition to allocation of input and output images. The implementation is multi-threaded, and processing one of the test inputs takes around 1-2 seconds on a quad-core processor.The class is tested to operate on both itk::Image and itk::RLEImage. Since all the processing is done on extracted slices, usage of itk::RLEImage for input and/or output affects performance to a limited degree.This class is implemented to ease manual segmentation in ITK-SNAP (www.itksnap.org). The class, along with test data and automated regression tests is packaged as an ITK remote module https://github.com/KitwareMedical/ITKMorphologicalContourInterpolation.

Automated Regression Testing Using Constraint Programming

In software validation, regression testing aims to check the absence of regression faults in new releases of a software system. Typically, test cases used in regression testing are executed during a limited amount of time and are selected to check a given set of user requirements. When testing large systems, the number of regression tests grows quickly over the years, and yet the available time slot stays limited. In order to overcome this problem, an approach known as test suite reduction (TSR), has been developed in software engineering to select a smallest subset of test cases, so that each requirement remains covered at least once. However solving the TSR problem is difficult as the underlying optimization problem is NP-hard, but it is also crucial for vendors interested in reducing the time to market of new software releases. In this paper, we address regression testing and TSR with Constraint Programming (CP). More specifically, we propose new CP models to solve TSR that exploit global constraints, namely NVALUE and GCC. We reuse a set of preprocessing rules to reduce a priori each instance, and we introduce a structureaware search heuristic. We evaluated our CP models and proposed improvements against existing approaches, including a simple greedy approach and MINTS, the state-of-theart tool of the software engineering community. Our experiments show that CP outperforms both the greedy approach and MINTS when it is interfaced with MiniSAT, in terms of percentage of reduction and execution time. When MINTS is interfaced with CPLEX, we show that our CP model performs better only on percentage of reduction. Finally, by working closely with validation engineers from Cisco Systems, Norway, we integrated our CP model into an industrial regression testing process.

Continuous Integration for Concurrent MOOSE Framework and Application Development on GitHub

For the past several years, Idaho National Laboratory’s MOOSE framework team has employed modern software engineering techniques (continuous integration, joint application/framework source code repos- itories, automated regression testing, etc.) in developing closed-source multiphysics simulation software (Gaston et al., Journal of Open Research Software vol. 2, article e10, 2014). In March 2014, the MOOSE framework was released under an open source license on GitHub, significantly expanding and diversifying the pool of current active and potential future contributors on the project. Despite this recent growth, the same philosophy of concurrent framework and application development continues to guide the project’s development roadmap. Several specific practices, including techniques for managing multiple repositories, conducting automated regression testing, and implementing a cascading build process are discussed in this short paper. Special attention is given to describing the manner in which these practices naturally synergize with the GitHub API and GitHub-specific features such as issue tracking, Pull Requests, and project forks.

ECCO version 4: an integrated framework for non-linear inverse modeling and global ocean state estimation

Abstract. This paper presents the ECCO v4 non-linear inverse modeling framework and its baseline solution for the evolving ocean state over the period 1992–2011. Both components are publicly available and subjected to regular, automated regression tests. The modeling framework includes sets of global conformal grids, a global model setup, implementations of data constraints and control parameters, an interface to algorithmic differentiation, as well as a grid-independent, fully capable Matlab toolbox. The baseline ECCO v4 solution is a dynamically consistent ocean state estimate without unidentified sources of heat and buoyancy, which any interested user will be able to reproduce accurately. The solution is an acceptable fit to most data and has been found to be physically plausible in many respects, as documented here and in related publications. Users are being provided with capabilities to assess model–data misfits for themselves. The synergy between modeling and data synthesis is asserted through the joint presentation of the modeling framework and the state estimate. In particular, the inverse estimate of parameterized physics was instrumental in improving the fit to the observed hydrography, and becomes an integral part of the ocean model setup available for general use. More generally, a first assessment of the relative importance of external, parametric and structural model errors is presented. Parametric and external model uncertainties appear to be of comparable importance and dominate over structural model uncertainty. The results generally underline the importance of including turbulent transport parameters in the inverse problem.

Automation Testing: An Introduction to Selenium

A good tool for testing is the one which has the ability to augment the testing performance and its competences as well as that can be a part of automation. When we chooses to build and host a Web application; automated regression testing becomes a responsibility for accomplishment. As the need of Web applications are growing, with each subsequent release, testing needs to grow tremendously. In this paper we will discuss the introduction to regression testing for Web applications and the Selenium testing framework. Selenium is open source and portable software available for different windows.

Influences on regression testing strategies in agile software development environments

Regression testing is a well-established practice in software development, but in recent years it has seen a change of status and emphasis with the increasing popularity of agile methods, which stress the central role of regression testing in maintaining software quality. The objectives of this article are to investigate regression testing strategies in agile development teams and identify the factors that can influence the adoption and implementation of this practice. We have used a mixed methods approach to our research, beginning with an analysis of the literature to identify research themes related to the adoption of regression testing techniques under agile methodologies, from which we developed an analytical framework for the study. This was followed by three exploratory case studies that we used to exercise the main elements of the framework, develop some key themes of interest, and devise a questionnaire for the final stage of the study, an on-line survey to explore the main issues identified in the case studies across different contexts. Within our specific sample, our results suggest that organizational maturity is a key factor in effective regression testing practices and that the adoption of such practices is helped by a coherent testing philosophy and change management processes. We also found that the return on investment in automated regression testing was positive for our respondents and that adopting these practices in the context of agile methods had been a relatively painless process for the organizations in our survey. We conclude that investing in regression testing tools and processes is likely to be beneficial for organizations. However, further work is needed in assessing how organizational culture impacts on the quality process and the financial outcomes for commercial software development organizations.

Research and Application of Automated Testing Tool Based on STAF

To improve the efficiency of automated testing, reduce testing spending, This article brings out a new automation testing frameworkWTAF, and automated regression testing is the focus of the study. First of all, we choose the open source automation framework STAF for regression testing , And then utilize STAF services to link all testing process implemented during the regression testing. Using automation testing technology, WTAF can successfully solve the low efficiency problem existed in manual or semi-automation testing.

Automated Regression Testing within Video Game Development

AbstractAutomated testing reduces the costs of software development. We propose a testing model that is specifically designed for the creation and execution of fully automated regression tests within video game development without the need of test isolation. The proposed model combines the usability and veracity of record and playback techniques and the possible test coverage of tests written in a game specific scripting language. The model has an intuitive structure that enables professional video game testers to create meaningful tests by using only a rudimentary set of programming skills within a graphical user interface. The resulting tool establishes a network connection with one or several concurrently running applications under test while the required performance can be distributed between several dedicated computers. The application under test is modified to process incoming function calls and to generate output that is used to dynamically control the course of the tests and to evaluate if the tested application operates within acceptable parameters.