Perform Regression Testing Research Articles

Performance regression testing and run-time verification of components in robotics systems

Unintended changes in the utilization of resources like CPU and memory can lead to severe problems for the operation of robotics and intelligent systems. Nevertheless, systematic testing for such performance regressions has largely been ignored in this domain. We present a method to specify and execute performance tests for individual components of component-based robotics systems based on their middleware interfaces. The method includes an automatic analysis of each component revision against previous ones that reports potential changes to the resource utilization characteristics. This informs developers about the impact of their changes. We describe the design of the framework and present evaluation results for the automatic detection of performance changes based on tests for a variety of robotics components. Additionally, we demonstrate how performance tests can be used as a basis for learning resource utilization models of components. These models can be used to detect faults at system run-time, which provides an additional level of safety for the systems besides the offline testing.

Time Based Prioritization of Test Cases in Regression Testing.

Background/Objectives: When client wants some change in functionality or in look/design of software, then after implementation of requirement, tester has to retest it. For this tester has to perform regression testing. It is done to verify that any other issue is not pushed into the system, and also verify existing issue if any, So, Regression testing not only is very important process when software change made but also it is very costly too. Methods: This paper proposes a new method based on ratio between number of issues found and run time of Test cases. This paper gives predilection to the original test suite so that the new suite, which is run within a time-constrained execution environment, will have a superior rate of fault detection. Findings: This paper presents a technique that prioritizes regression test cases so that the new ordered test case produces a high productivity. Application/Improvements: This paper assumes data for pre-executed test cases in the form of number of issues found and run time.

Test case prioritisation during web application testing

Owing to frequent alterations in the existing web applications, performing regression testing becomes necessary for the identification and rectification of the newly generated unwanted faults. Owing to various resource constraints, test case prioritisation is one of the strategies followed, rather than running test cases blindly. This paper proposes a novel approach towards prioritisation of test cases during regression testing of web application (dynamic website) using Bayesian network. Initially, a Bayesian network is formed using various parameters which affect the success of a test case as well as promote testing of more crucial sections of the web application. Thereafter, the conditional probability table and probabilistic inference algorithms are applied to evaluate the success probability and ultimately priority (importance) of a test case. Execution of the test cases takes place on the basis of their respective priority. The performance of proposed technique is compared with existing work, 2-opt inspired heuristic algorithm and genetic algorithm.

Test case prioritisation during web application testing

Owing to frequent alterations in the existing web applications, performing regression testing becomes necessary for the identification and rectification of the newly generated unwanted faults. Owing to various resource constraints, test case prioritisation is one of the strategies followed, rather than running test cases blindly. This paper proposes a novel approach towards prioritisation of test cases during regression testing of web application (dynamic website) using Bayesian network. Initially, a Bayesian network is formed using various parameters which affect the success of a test case as well as promote testing of more crucial sections of the web application. Thereafter, the conditional probability table and probabilistic inference algorithms are applied to evaluate the success probability and ultimately priority (importance) of a test case. Execution of the test cases takes place on the basis of their respective priority. The performance of proposed technique is compared with existing work, 2-opt inspired heuristic algorithm and genetic algorithm.

A Multi-Objective Technique to Prioritize Test Cases

While performing regression testing, an appropriate choice for test case ordering allows the tester to early discover faults in source code. To this end, test case prioritization techniques can be used. Several existing test case prioritization techniques leave out the execution cost of test cases and exploit a single objective function (e.g., code or requirements coverage). In this paper, we present a multi-objective test case prioritization technique that determines the ordering of test cases that maximize the number of discovered faults that are both technical and business critical. In other words, our new technique aims at both early discovering faults and reducing the execution cost of test cases. To this end, we automatically recover links among software artifacts (i.e., requirements specifications, test cases, and source code) and apply a metric-based approach to automatically identify critical and fault-prone portions of software artifacts, thus becoming able to give them more importance during test case prioritization. We experimentally evaluated our technique on 21 Java applications. The obtained results support our hypotheses on efficiency and effectiveness of our new technique and on the use of automatic artifacts analysis and weighting in test case prioritization.

Test Case Prioritization for Optimizing a Regression Test

Regression testing makes sure that upgradation of software in terms of adding new features or for bug fixing purposes should not hamper previously working functionalities. Whenever a software is upgraded or modified, a set of test cases are run on each of its functions to assure that the change to that function is not affecting other parts of the software that were previously running flawlessly. For achieving this, all existing test cases need to run as well as new test cases might be required to be created. It is not feasible to reexecute every test case for all the functions of a given software, because if there is a large number of test cases to be run, then a lot of time and effort would be required. This problem can be addressed by prioritizing test cases. Test case prioritization technique reorders the priority in which test cases are implemented, in an attempt to ensure that maximum faults are uncovered early on by the high priority test cases implemented first. In this paper we propose an optimized test case prioritization technique using Ant Colony Optimization (ACO) to reduce the cost, effort and time taken to perform regression testing and also uncover maximum faults. Comparison of different techniques such as Retest All, Test Case Minimization, Test Case Prioritization, Random Test Case Selection and Test Case Prioritization using ACO is also depicted.

Session details: Special Issue on Challenges in Software Performance

Software systems have become increasingly complex. Such complexity makes it extremely challenging to deliver systems that meet their performance requirements. In recent years, there have been several high profile instances of projects that failed due to performance-related issues, e.g., the rollout of the healthcare.gov Web site. This motivates the need for new approaches for software developers to address performance problems. This special issue presents 6 papers that address this theme. A brief summary of these papers follows. Gesvindr and Buhnova study the challenges of building Platform as a Service (PaaS) cloud applications that meet their performance requirements. Using two case studies involving the Microsoft Azure cloud platform, the authors show that the use of a cloud platform in itself does not guarantee good performance and scalability. The authors then identify several best practices for ensuring the performance and scalability of PaaS applications. The work of Heinrich considers the problem of managing\ the performance of software systems composed of thirdparty cloud services. The lack of control over such thirdparty services can cause deviations from performance objectives that can be hard to foresee and manage. The author outlines a model-based framework called iObserve that can help a system adapt and evolve so as to address such problems. Li, O'Brien, and Kihl address the lack of standard methodologies for experimental performance evaluation of software systems. The paper outlines a method for defining domainspecific methodologies for performance evaluation in a way that is consistent across domains. The authors demonstrate the utility of their approach using a cloud services case study. The paper by Grottke et al. considers the problem of handling cyber attacks on networked computer systems andconnected industrial control systems supporting critical infrastructuresuch as energy, water, and gas. Using analytic models, the authors study the performance of various attack mitigation approaches. In particular, the model shows how parameters such as malware scanning rates, network topology, and types of infections impact the effective processing capacity of a system. Rehmann et al. describe the performance monitoring in SAP HANA's continuous integration process. They describe how performance regression tests have been integrated in the delivery process and what services are available to detect and report performance anomalies. Being an extended version of a previous workshop paper, this paper now also presents the reporting of performance test results and anomaly detection. Finally, Nambiar et al. examine issues and experience with software performance, and consider the shortcomings and potential benefits of modeling from the perspective of a potential industrial user. In particular, it considers why modeling is little used, and what conditions it would have to satisfy to fulfill its potential. We sincerely thank all authors for their work and contribution to this special issue. Additionally, we thank our reviewers Davide Arcelli, Danilo Ardagna, Alberto Avritzer, Steffen Becker, Ian Gorton, Andre van Hoorn, Raffaela Mirandola, Juan F. Perez, and Murray Woodside for their timely and constructive reviews as well as editor-in-chief Nidhi Hedge for her support.

Optimized Regression Test Using Test Case Prioritization

Regression testing ensures that changes made in the fixes or any enhancement changes do not impact the previously working functionality. Whenever software is modified, a set of test cases are run to assure that changes don’t affect the other parts of the software. Hence all existing test cases need to be tested as well as new test cases need to be created. It is nonviable to re-execute every test case for a given software, because if there are more number of test cases to be tested, the more effort and time is required. This problem can be solved by prioritizing test cases. Test case prioritization techniques reorder the priority of a test case in an attempt to ensure that maximum faults are uncovered by the high prioritized test cases. In this paper we propose an optimized test case prioritization technique using Ant Colony Optimization (ACO) to reduce the cost, effort and time taken to perform regression testing and also uncover maximum faults.

PENGARUH STRUKTUR KEPEMILIKAN, STRUKTUR MODAL, DAN UKURAN PERUSAHAAN TERHADAP NILAI PERUSAHAAN PADA PERUSAHAAN SEKTOR PERTAMBANGAN YANG TERDAFTAR DI BURSA EFEK INDONESIA

This study aims to determine whether there is influence of managerial ownership structure, institutional ownership structure, capital structure, and firm size to company value in mining sector companies listed in Indonesia Stock Exchange.The method of this research is descriptive method used to analyze data by way of describing or giving description to the object under study through sample data or population as it is without doing analysis and make conclusion which apply to public. This research uses managerial ownership variable, institutional ownership, capital structure, and firm size as independent variable, and firm value as dependent variable. The population of this study is a mining sector company listed on the Indonesia Stock Exchange in 2010-2012. After going through purposive sampling, obtained 29 companies as sample. The type of data used in this study is secondary data in the form of annual financial statements of the company. The method of analysis used is simple linear regression analysis. Before performing regression test, data analysis test consisted of linearity test and normality test.The results of this study indicate that managerial ownership, capital structure and firm size have a positive effect on firm value. While institutional ownership variable has no positive effect on firm value.

Anomaly detection in performance regression testing by transaction profile estimation

SummaryAs part of the process to test a new release of an application, the performance testing team need to confirm that the existing functionalities do not perform worse than those in the previous release, a problem known as performance regression anomaly. Most existing approaches to analyse performance regression testing data vary according to the applied workload, which usually leads to the need for an extra performance testing run. To ease such lengthy tasks, we propose a new workload‐independent, automated technique to detect anomalies in performance regression testing data using the concept known as transaction profile (TP). The TP is inferred from the performance regression testing data along with the queueing network model of the testing system. Based on a case study conducted against two web applications, one open source and one industrial, we have been able to automatically generate the ‘TP run report’ and verify that it can be used to uncover performance regression anomalies caused by software updates. In particular, the report helped us to isolate the real anomaly issues from those caused by workload changes with an average F1 measure of 85% for the open source application and 90% for the industrial application. Such results support our proposal to use the TP as a more efficient technique in identifying performance regression anomalies than the state of the art industry and research techniques. Copyright © 2015 John Wiley & Sons, Ltd.

Distance Based Root Cause Analysis and Change Impact Analysis of Performance Regressions

Performance regression testing is applied to uncover both performance and functional problems of software releases. A performance problem revealed by performance testing can be high response time, low throughput, or even being out of service. Mature performance testing process helps systematically detect software performance problems. However, it is difficult to identify the root cause and evaluate the potential change impact. In this paper, we present an approach leveraging server side logs for identifying root causes of performance problems. Firstly, server side logs are used to recover call tree of each business transaction. We define a novel distance based metric computed from call trees for root cause analysis and apply inverted index from methods to business transactions for change impact analysis. Empirical studies show that our approach can effectively and efficiently help developers diagnose root cause of performance problems.

Hybrid regression testing technique: based on requirement priorities, fault and modification history

Context: The main challenge in incremental delivery of software is the testing of the increment with minimal number of test cases in such a manner that effect of execution of the subset is equivalent to the execution of whole test suite regression testing. The execution of test suite subset must guarantee the capability of revealing all faults at the earliest. Objective: This paper aims to perform regression testing with minimal number of test cases that are capable of locating large percentage of faults. Method: The hybrid regression testing algorithm proposed in this paper is based on the clustering by employing three parameters: a requirement priorities; b fault detection history; c module modification history. Results: The proposed technique was applied on live project of Tool for automatic analysis and comparison of different release planning methods. The total of 14 requirements are considered for evaluation of proposed testing technique three were added after the first review, leading to the process of reprioritisation. Conclusion: The evaluation of technique on real software revealed that the proposal resulted in better faults localisation with minimal number of test cases although its evaluation on a mature software system is kept as future work.

An efficient regression testing approach for PHP web applications: a controlled experiment

SUMMARYCompanies that provide web applications often encounter various security attacks and frequent feature‐update demands from users, and when these needs arise, companies need to fix security problems or upgrade the application with new features. These fixes often involve small patches or revisions, but still, testers need to perform regression testing on their products to ensure that the changes have not introduced new faults. Performing regression testing on the entire product, however, can be very expensive, and it is not a viable solution for companies that need a short turnaround time to release patches. One solution is focusing only on the code areas that have been changed and performing regression testing on them. By doing this, companies can provide quick patches more dependably whenever they encounter security breaches. In this paper, the authors proposed a new regression testing approach that identifies the affected areas by code changes using impact analysis and generates new test cases for the impacted areas by changes using program slices. To facilitate the approach, the researchers implemented a Hypertext Preprocessor (PHP) Analysis and Regression Testing Engine (PARTE) and performed a controlled experiment using five open source web applications with multiple versions. The results showed that this approach is effective in reducing the cost of regression testing for a frequently patched web application, and exposed ways in which that effectiveness can vary with application characteristics and versioning frequencies. Copyright © 2014 John Wiley & Sons, Ltd.

Near Real-time Data Analysis of Core-collapse Supernova Simulations with Bellerophon

We present an overview of a software system, Bellerophon, built to support a production-level HPC application called CHIMERA, which simulates core-collapse supernova events at the petascale. Developed over the last four years, Bellerophon enables CHIMERA's geographically dispersed team of collaborators to perform data analysis in near real-time. Its n-tier architecture provides an encapsulated, end-to-end software solution that enables the CHIMERA team to quickly and easily access highly customizable animated and static views of results from anywhere in the world via a web- deliverable, cross-platform desktop application. In addition, Bellerophon addresses software engineering tasks for the CHIMERA team by providing an automated mechanism for performing regression testing on a variety of supercomputing platforms. Elements of the team's workflow management needs are met with software tools that dynamically generate code repository statistics, access important online resources, and monitor the current status of several supercomputing resources.

A formal approach for run-time verification of web applications using scope-extended LTL

ContextIn the past decade, the World Wide Web has been subject to rapid changes. Web sites have evolved from static information pages to dynamic and service-oriented applications that are used for a broad range of activities on a daily basis. For this reason, thorough analysis and verification of Web Applications help assure the deployment of high quality applications. ObjectivesIn this paper, an approach is presented to the formal verification and validation of existing web applications. The approach consists of using execution traces of a web application to automatically generate a communicating automata model. The obtained model is used to model checking the application against predefined properties, to perform regression testing, and for documentation. MethodsTraces used in the proposed approach are collected by monitoring a web application while it is explored by a user or a program. An automata-based model is derived from the collected traces by mapping the pages of the application under test into states and the links and forms used to browse the application into transitions between the states. Properties, meanwhile, express correctness and quality requirements on web applications and might concern all states of the model; in many cases, these properties concern only a proper subset of the states, in which case the model is refined to designate the subset of the global states of interest. A related problem of property specification in Linear Temporal Logic (LTL) over only a subset of states of a system is solved by means of specialized operators that facilitate specifying properties over propositional scopes in a concise and intuitive way. Each scope constitutes a subset of states that satisfy a propositional logic formula. ResultsAn implementation of the verification approach that uses the model checker Spin is presented where an integrated toolset is developed and empirical results are shown. Also, Linear Temporal Logic is extended with propositional scopes. Conclusiona formal approach is developed to build a finite automata model tuned to features of web applications that have to be validated, while delegating the task of property verification to an existing model checker. Also, the problem of property specification in LTL over a subset of the states of a given system is addressed, and a generic and practical solution is proposed which does not require any changes in the system model by defining specialized operators in LTL using scopes.

A Systematic Review of Techniques for Test Case Prioritization

In software development life cycle, maintenance phase is an important phase as it deals with the activities like error correction, upgradation, deletion and optimization of software capabilities. For this reason, regression testing is required in order to revalidate the modifications in the software. It is an expensive process. Various techniques of performing regression testing are available. Software testers can select the technique that suit their requirement as well as optimize the basic cost and time factors. This paper mainly discusses various test case prioritization techniques for regression testing presented by various researchers and the various search algorithms used in the test case prioritisation process.

STABILIZER

Researchers and software developers require effective performance evaluation. Researchers must evaluate optimizations or measure overhead. Software developers use automatic performance regression tests to discover when changes improve or degrade performance. The standard methodology is to compare execution times before and after applying changes. Unfortunately, modern architectural features make this approach unsound. Statistically sound evaluation requires multiple samples to test whether one can or cannot (with high confidence) reject the null hypothesis that results are the same before and after. However, caches and branch predictors make performance dependent on machine-specific parameters and the exact layout of code, stack frames, and heap objects. A single binary constitutes just one sample from the space of program layouts, regardless of the number of runs. Since compiler optimizations and code changes also alter layout, it is currently impossible to distinguish the impact of an optimization from that of its layout effects. This paper presents Stabilizer, a system that enables the use of the powerful statistical techniques required for sound performance evaluation on modern architectures. Stabilizer forces executions to sample the space of memory configurations by repeatedly re-randomizing layouts of code, stack, and heap objects at runtime. Stabilizer thus makes it possible to control for layout effects. Re-randomization also ensures that layout effects follow a Gaussian distribution, enabling the use of statistical tests like ANOVA. We demonstrate Stabilizer's efficiency (<7% median overhead) and its effectiveness by evaluating the impact of LLVM's optimizations on the SPEC CPU2006 benchmark suite. We find that, while -O2 has a significant impact relative to -O1, the performance impact of -O3 over -O2 optimizations is indistinguishable from random noise.

STABILIZER

Researchers and software developers require effective performance evaluation. Researchers must evaluate optimizations or measure overhead. Software developers use automatic performance regression tests to discover when changes improve or degrade performance. The standard methodology is to compare execution times before and after applying changes. Unfortunately, modern architectural features make this approach unsound. Statistically sound evaluation requires multiple samples to test whether one can or cannot (with high confidence) reject the null hypothesis that results are the same before and after. However, caches and branch predictors make performance dependent on machine-specific parameters and the exact layout of code, stack frames, and heap objects. A single binary constitutes just one sample from the space of program layouts, regardless of the number of runs. Since compiler optimizations and code changes also alter layout, it is currently impossible to distinguish the impact of an optimization from that of its layout effects. This paper presents Stabilizer, a system that enables the use of the powerful statistical techniques required for sound performance evaluation on modern architectures. Stabilizer forces executions to sample the space of memory configurations by repeatedly re-randomizing layouts of code, stack, and heap objects at runtime. Stabilizer thus makes it possible to control for layout effects. Re-randomization also ensures that layout effects follow a Gaussian distribution, enabling the use of statistical tests like ANOVA. We demonstrate Stabilizer's efficiency (<7% median overhead) and its effectiveness by evaluating the impact of LLVM's optimizations on the SPEC CPU2006 benchmark suite. We find that, while -O2 has a significant impact relative to -O1, the performance impact of -O3 over -O2 optimizations is indistinguishable from random noise.

A Performance Anomaly Detection and Analysis Framework for DBMS Development

Detecting performance anomalies and finding their root causes are tedious tasks requiring much manual work. Functionality enhancements in DBMS development as in most software development often introduce performance problems in addition to bugs. To detect the problems as soon as they are introduced, which often happens during the early phases of a development cycle, we adopt performance regression testing early in the process. In this paper, we describe a framework that we developed to manage performance anomalies after establishing a set of conditions for a problem to be considered an anomaly. The framework uses Statistical Process Control (SPC) charts to detect performance anomalies and differential profiling to identify their root causes. By automating the tasks within the framework we were able to remove most of the manual overhead in detecting anomalies and reduce the analysis time for identifying the root causes by about 90 percent in most cases. The tools developed and deployed based on the framework allow us continuous, automated daily monitoring of performance in addition to the usual functionality monitoring in our DBMS development.

A safe regression testing approach for safety critical systems

Regression testing is important activity during the software maintenance to deal with adverse effects of changes. Our approach is important for safety critical system as usually formal methods are preferred and highly recommended for the safety critical systems but they are also applied for the systems development of other than critical system. Our approach is based on Regression testing using VDM++ which takes two VDM++ specifications, one baseline and other delta (Changed) along with test suite for the baseline version. It compares both versions by using comparator module, identifies the change. By analyzing the change we classify the test cases from original test suite into obsolete, re-testable, and reusable test cases. Our scope is at unit level i.e. at class level. Our approach gets two versions of VDM++ specification and returns regression test suite for the delta version. Our approach distinguishes test cases which are still effective for the delta version of VDM++ specification and it differs from re-test all strategy as it can distinguish the test cases and identifies test cases which are useful for delta version. Test cases reusability and test case reduction is the main objective of our approach. Our approach presents how to perform regression testing using VDM++ specification during the maintenance of systems.