Problem In Software Testing Research Articles

РАЗРАБОТКА МЕТОДИКИ ВЫЯВЛЕНИЯ СЛЕПЫХ ЗОН ПРИ ТЕСТИРОВАНИИ ПРОГРАММНОГО ОБЕСПЕЧЕНИЯ

The article proposes a method that will help identify blind spots in software testing. Common problems in software testing and methods for their solution are described. A method to detect a problem is presented. The study also demonstrates an example of implementing this method.

Fuzzing vs SBST

Search-Based Software Testing (SBST) is the application of SBSE for solving hard software testing problems. SBST has been the subject of discussion of our International SBST Workshop, colocated with the International Conference on Software Engineering (ICSE). In 2022 we hosted the 15th edition of our SBSTWorkshop, which brought together researchers and industrial practitioners to encourage the use of search-based and fuzz testing techniques and tools for addressing software engineering-specific challenges. In this 2022 edition, SBST held, among other exciting events, a discussion panel on the similarities and differences between SBST and Fuzzing. As it implies, the goal of the panel was to have a common ground for discussion on the main similarities and differences between the Fuzzing and SBST fields, focusing on how both communities can collaborate to advance the stateof- the-art on automated testing. This strong panel composed of researchers from both academia and industry was the highlight of SBST'22 and allowed the chairs of the workshop to make substantial changes for 2023. In this paper, we present the main take away messages from that seminal panel and highlight exciting new challenges in the field.

Generating of Test Data by Harmony Search Against Genetic Algorithms

Many search-based algorithms have been successfully applied in several software engineering activities. Genetic algorithms (GAs) are the most used in the scientific domains by scholars to solve software testing problems. They imitate the theory of natural selection and evolution. The harmony search algorithm (HSA) is one of the most recent search algorithms in the last years. It imitates the behavior of a musician to find the best harmony. Scholars have estimated the similarities and the differences between genetic algorithms and the harmony search algorithm in diverse research domains. The test data generation process represents a critical task in software validation. Unfortunately, there is no work comparing the performance of genetic algorithms and the harmony search algorithm in the test data generation process. This paper studies the similarities and the differences between genetic algorithms and the harmony search algorithm based on the ability and speed of finding the required test data. The current research performs an empirical comparison of the HSA and the GAs, and then the significance of the results is estimated using the t-Test. The study investigates the efficiency of the harmony search algorithm and the genetic algorithms according to (1) the time performance, (2) the significance of the generated test data, and (3) the adequacy of the generated test data to satisfy a given testing criterion. The results showed that the harmony search algorithm is significantly faster than the genetic algorithms because the t-Test showed that the p-value of the time values is 0.026 < α (α is the significance level = 0.05 at 95% confidence level). In contrast, there is no significant difference between the two algorithms in generating the adequate test data because the t-Test showed that the p-value of the fitness values is 0.25 > α.

Tabular‐expression‐based method for constructing metamorphic relations

SummaryMetamorphic testing (MT) is proposed to overcome the oracle problem in software testing, and metamorphic relations (MRs) are the core of MT. There is a lack of guidelines for constructing effective MRs, and it is difficult to reuse MRs mainly because most MRs are closely related to the domain knowledge. In this article, we propose a method for constructing MRs from specifications in tabular expression format. Our method constructs MRs according to the characteristics of tabular expressions, especially the relationships between the header grids and the main grid, namely, our method is domain‐independent and the construction process is simplified. In addition, the derived MRs can be applied to specifications with the same tabular expression structure. For specifications with different tabular expression structures, MRs can still be used after slight adjustments. To evaluate the performance of our method in practice, we apply the method to five applications. The experimental results demonstrate that our method is effective for a program with the oracle problem, and that it is applicable to tabular expressions in various formats. Compared with representative testing methods, our method identifies errors that are not detected by the compared methods. Hence, our method and existing methods can complement each other. The MR proposed in this article outperforms MRs constructed based on program properties.

An Empirical Comparison of the Efficiency and Effectiveness of Genetic Algorithms and Adaptive Random Techniques in Data-Flow Testing

Software Testing depends on the execution of the tested-program against a set of test-inputs and the comparison of its outputs with the expected ones. The size of the input domain is very large that can be the set of real numbers ( $\mathbb {R}$ ). Thus, the selection of the appropriate inputs is one of the key problems in software testing. This process is time-consuming and needs a lot of effort and budget. Therefore, automatic inputs generation techniques are required to overcome these problems. Genetic algorithms (GAs) have been successfully used for generating test-inputs. Researchers proved that GAs overcame ordinary random search techniques (ORTs) in generating inputs. In addition, GAs can converge faster than ordinary random techniques and they can reduce effectively the size of the test-suite. Unfortunately, technically GAs needs time more than ORTs. Adaptive random testing technique (ART) is a form of ORT that works for distributing test-cases more evenly through the input domain to increase the efficiency of ORTs. So far, there is no study comparing the efficiency of GAs and ARTs in data-flow testing. In this paper, we introduce an empirical comparison for genetic algorithms and adaptive random techniques according to four factors: reducing the size of the test-suite, convergence speed, elapsed time, and the effectiveness in maximizing the coverage ratio of all du-pairs criterion. The experimental study, which was conducted to compare the two techniques, contains 7 Java programs. The results of the experiments showed that the GA technique defeated the ORT technique and the ART technique in reducing the size of the required test-suite to satisfy all du-pairs criterion. Where the GA technique created in total 31532 test-inputs while the ART technique generated 61841 and the ORT technique produced 32064. Further, the results showed that the GA technique converged faster than the ORT technique and the ART technique. Where the procedure of the GA technique was repeated 3153 times totally while the procedure of the ART technique was iterated 6184 times and the procedure of the ORT technique was repeated 3206 times. In addition, the convergence rate of GA-based technique = 8.25 generations/second, the convergence rate of the ORT = 11.98 generations/second, and the convergence rate of the ART = 13.27 generations/second. Moreover, the results showed that the GA technique is faster than the ART technique and slower than the ORT technique. Where the GA technique consumed in total 382 seconds while the ART technique consumed 465.9 seconds and the ORT technique consumed 267.6 seconds. Additionally, the results showed that the GA technique satisfied overall coverage ratio equals 74% of all du-pairs while the ART technique satisfied 78% and the ORT technique satisfied 73%. From these results, we concluded that GA algorithms are more effective than ORT and ART techniques in data-flow testing but ART satisfied the most coverage ratio. Therefore, we recommend hybridizing GA and ART and applying the hybrid technique in the test-data generation process.

An iterative metamorphic testing technique for web services and case studies

Metamorphic testing (MT) is an innovative approach to alleviating the oracle problem in software testing, which uses metamorphic relations of the program under test, instead of the test oracles, to verify its outputs. To alleviate the oracle problem of testing web services, we had previously proposed an MT framework for web services. In this paper, we further improve the efficiency and automation of this framework by leveraging metamorphic relations to iteratively generate test cases. We present a fixed-size iterative MT algorithm and implement it in the MT framework. We conduct three case studies to evaluate the fault detection effectiveness and efficiency of the proposed approach. Experimental results suggest that, compared with the conventional MT, iterative MT can achieve a comparable fault detection effectiveness, but with significantly fewer resources. Observations and limitations are summarised to provide new insights into the application of iterative MT.

Semiautomated Metamorphic Testing Approach for Geographic Information Systems: An Empirical Study

A geographic information system (GIS) provides basic location-enabled services for many different applications related to navigation, education, and telecommunications. It is a foundation for analysis and visualization. Testing GIS is critical, but challenging due to the difficulty to assess the correctness of GIS outputs, which is called the test oracle problem of software testing. Metamorphic testing alleviates the problem by constructing metamorphic relations (MRs) among multiple inputs and outputs of the program under test. In this article, a semiautomated metamorphic testing (SAMT) method, based on the formal MR model and an improved adaptive random testing algorithm, was proposed to the GIS. To evaluate the performance of our approach, we conducted a case study on a superficial area calculation program, a typical component of GIS. Six kinds of MR construction methods were suggested for the GIS domain program testing. The experimental results show that SAMT can detect the mutations effectively that could solve the test oracle problem efficiently. More importantly, there is no need to manual participation in the testing process, except for the MR construction.

METRIC+: A Metamorphic Relation Identification Technique Based on Input plus Output Domains

Metamorphic testing is well known for its ability to alleviate the oracle problem in software testing. The main idea of metamorphic testing is to test a software system by checking whether each identified metamorphic relation (MR) holds among several executions. In this regard, identifying MRs is an essential task in metamorphic testing. In view of the importance of this identification task, METRIC (METamorphic Relation Identification based on Category-choice framework) was developed to help software testers identify MRs from a given set of complete test frames. However, during MR identification, METRIC primarily focuses on the input domain without sufficient attention given to the output domain, thereby hindering the effectiveness of METRIC. Inspired by this problem, we have extended METRIC into METRIC $^{+}$ + by incorporating the information derived from the output domain for MR identification. A tool implementing METRIC $^{+}$ + has also been developed. Two rounds of experiments, involving four real-life specifications, have been conducted to evaluate the effectiveness and efficiency of METRIC $^{+}$ + . The results have confirmed that METRIC $^{+}$ + is highly effective and efficient in MR identification. Additional experiments have been performed to compare the fault detection capability of the MRs generated by METRIC $^{+}$ + and those by $\mu$ μ MT (another MR identification technique). The comparison results have confirmed that the MRs generated by METRIC $^{+}$ + are highly effective in fault detection.

A greedy particle swarm optimization (GPSO) algorithm for testing real-world smart card applications

Software testing continues to be regarded as a necessary and critical step in the software development life cycle. Among the multitudes of existing techniques, particle swarm optimization (PSO) algorithm, in particular, has shown superior merits for automatically generating software test cases for its easy implementation and for relying on fewer parameters that require tuning. Hence, several state-of-the-art PSO-based algorithms have been successfully used as a test data generator. On the other hand, greedy-based algorithms, which are commonly used to solve complex and multi-step combinatorial problems, are starting to gain momentum as a solution for the complexity problem of software testing. Greedy algorithms favored over other techniques when the solution of the problem is guaranteed to be near-optimal. As a result, the utilization of both greedy and PSO algorithms in a single solution for automatically generating test data represents a strong candidate if designed carefully. In this paper, we propose a novel hybrid greedy and PSO algorithm (GPSO) that jointly guarantees the effectiveness and close to optimality results for generating a minimum number of test data. Compared with the widely employed genetic algorithm (GA), our proposed GPSO outperforms the GA in terms of the average number of iterations, execution time, and coverage percentage. Experimental trials with six different typical Java card applications show that the use of the proposed GPSO as a test data generator results in an outstanding performance.

Multi-trajectory Fitness Function-based Method of Automated Test Coverage for Code Using Evolutionary Algorithms

Testing is a time consuming and resource intensive task. Testing often takes about half of the total project development time.Hence, in recent times, a lot of studies have considered the issues of the process automation of creating and running tests. Recently, a lot of papers have been published in the field concerned. However, most of them use unit testing, in which the coverage of different program fragments occurs independently of each other. Therefore, the internal structure of the program was not taken into account.The article presents a method to build a test suite based on evolutionary algorithms that takes into account the features of the entire program, as a whole.Evolutionary algorithms are a class of optimization algorithms that has been widely developed lately, using the ideas of artificial intelligence and intended for conducting directed search. They allow successful solving the problems of software testing, such as testing embedded systems and automatic construction of unit tests.With implementing evolutionary approach the article has solved the following tasks. Proposed a method for determining the fitness function, which allows us to consider an approximation along all possible trajectories, which provides a solution to the problem of coverage in almost any case. However, the number of such trajectories can be quite large, which leads to a significant increase in test execution time. Proposed a method of approximation along all possible trajectories, allowing testing to be done in a reasonable time. Time costs to achieve the goal are reduced by selecting such a trajectory among all possible ones for which a distance value to the instruction is minimal.To study the proposed method, there were four different tasks each of which involved about 1000 runs. The presented research results have shown that the given method is more reliable than the commonly used approaches in which the reference trajectory is selected. In addition, an estimate of time costs has shown that the discussed method has higher efficiency.

Optimizing Cost Function in Imperialist Competitive Algorithm for Path Coverage Problem in Software Testing

Search-based optimization methods have been used for software engineering activities such as software testing. In the field of software testing, search-based test data generation refers to application of meta-heuristic optimization methods to generate test data that cover the code space of a program. Automatic test data generation that can cover all the paths of software is known as a major challenge. The paper establishes a new cost function for automatic test data generation, which can traverse the non-iterative paths of software control flow graphs. This function is later compared with similar cost functions proposed in other articles. The results indicate the superior performance of the proposed function. Still another innovation in this paper is the application of the Imperialist Competitive Algorithm in automatic test data generation along with the proposed cost function. Automatic test data generation is implemented through the Imperialist Competitive Algorithm as well as the Genetic and Particle Swarm Optimization Algorithms for three software programs with different search space sizes. The algorithms are compared with each other in terms of convergence speed, computational time, and local search. Test data generated by the proposed method has achieved better results than other algorithms in finding the number of non-iterative paths, the convergence speed and computational time with growing the searching space of the software's control flow graph.

Transferring interactive search-based software testing to industry

Context: Search-Based Software Testing (SBST), and the wider area of Search-Based Software Engineering (SBSE), is the application of optimization algorithms to problems in software testing, and software engineering, respectively. New algorithms, methods, and tools are being developed and validated on benchmark problems. In previous work, we have also implemented and evaluated Interactive Search-Based Software Testing (ISBST) tool prototypes, with a goal to successfully transfer the technique to industry.Objective: While SBST and SBSE solutions are often validated on benchmark problems, there is a need to validate them in an operational setting, and to assess their performance in practice. The present paper discusses the development and deployment of SBST tools for use in industry, and reflects on the transfer of these techniques to industry.Method: In addition to previous work discussing the development and validation of an ISBST prototype, a new version of the prototype ISBST system was evaluated in the laboratory and in industry. This evaluation is based on an industrial System under Test (SUT) and was carried out with industrial practitioners. The Technology Transfer Model is used as a framework to describe the progression of the development and evaluation of the ISBST system, as it progresses through the first five of its seven steps.Results: The paper presents a synthesis of previous work developing and evaluating the ISBST prototype, as well as presenting an evaluation, in both academia and industry, of that prototype’s latest version. In addition to the evaluation, the paper also discusses the lessons learned from this transfer.Conclusions: This paper presents an overview of the development and deployment of the ISBST system in an industrial setting, using the framework of the Technology Transfer Model. We conclude that the ISBST system is capable of evolving useful test cases for that setting, though improvements in the means the system uses to communicate that information to the user are still required. In addition, a set of lessons learned from the project are listed and discussed. Our objective is to help other researchers that wish to validate search-based systems in industry, and provide more information about the benefits and drawbacks of these systems.

Research on the Improvement of Aerospace Software Test Model Based on V Model

The software industry has endured a long and tortuous development process from the beginning to the present, from small scale, low complexity of software development to the large scale of application of software products, software testing plays an important role, the early software debugging has been evolved into software testing now. With the rapid development of space industry in China, the position of software in it has been more and more important, software testing is very significant throughout the software development process, therefore software testing needs more concern. Based on the theory of software testing in software engineering, combined with the practice in work, the current problems in aerospace software testing, how to find existing problems and new testing methods in order to improve the quality of aerospace software. The traditional V model requires that each testing phase could proceed only after the previous stage completed, and there is no backtracking. We cannot adjust and modify the model when problems are found. In order to overcome the shortcomings of traditional V model, the paper puts forward an improved method of aerospace software model based on V model. By iterating over the various stages testings of software development, this method enhances intersections and comprehensiveness, so as to achieve the aim of reducing the number of testing problems and providing effective solutions for the aerospace software testing problems.

DENSITY BASED SPATIAL CLUSTERING DEVIATIONS FOR BLACK BOX REGRESSION TESTING IN LARGE DATABASE

Ensuring the functional excellence of database applications is a most needed and important problem in software testing. Database applications are mostly adopted in many fields. For example in public administrations they need to process large amounts of transactions efficiently and need to store large amounts of data. Regression testing is used to check whether the new changes cause any errors in the existing software. Regression testing is one of the most useful software testing types during software maintenance. Clustering is a data mining technique used to discover patterns from the database. This research work incorporates soft clustering concept, which is the process of deriving the information based on the similarity of the unsupervised database. It can be considered the most important unsupervised learning technique and it deals with finding a structure in a collection of unlabeled data. Black-box Regression Testing approach focuses on regression testing and proactively exposes behavioral deviations by checking inside the black box instead of checking only black-box outputs. The proposed research work introduces a new approach called Expectation-Maximization (EM) with Density Based Spatial Clustering Algorithm to measure the dissimilarity of data elements in large database. This research work, presents an optimal perspective on the problem of EM Clustering Deviations for Black Box Regression Testing of Database Applications. The proposed method called Density based spatial clustering deviations for black box regression testing in large database which measures the correspondence between pairs of data points. The proposed method is to establish a unified framework on unsupervised data sets. To validate the proposed approach, a large scale Norwegian Tax Accounting System case study is considered, and the results show that clustering approach can indeed serve as an accurate strategy for grouping regression test deviations. The proposed analysis suggests that this approach can significantly reduce the effort spent by testers in analyzing regression test deviations, increase their level of confidence, and therefore make regression testing more scalable.

ET-based Test Data Generation for Multiple-path Testing

Automatic test data generation for path testing is one of the difficult problems of software testing And the existing methods have low efficiency and stability. We have conducted studies on the research field about ET-based test data generation for multiple-path testing, and we have improved the efficiency and the stability of the method by improved the fitness function of currently existing method. By the comparison experiments, it showed that the method in this paper have high efficiency and stability. Meanwhile, we gave the conclusion how the basic parameters impacted on the efficiency, and suggested the setting range of parameters.

Experimental Analysis of m-ACO Technique for Regression Testing

Objectives: Experimental evaluation of “m-ACO” (Modified Ant Colony Optimization) technique for test case prioritization has been performed on two well known software testing problems namely “Triangle Classification Problem” and “Quadratic Equation Problem”. Apart from these two problems, m-ACO has been experimentally evaluated using open source software JFreeChart. Methods: m-ACO finds the optimized solution to test suite prioritization by modifying the phenomenon used by natural ants to reach to its food source and select the food. This paper attempts to experimentally and comparatively evaluate the proposed m-ACO technique for test case prioritization against some contemporary meta-heuristic techniques using two well known software testing problems and open source problem. Performance evaluation has been measured using two metrics namely APFD (Average Percentage of Faults Detected) and PTR (Percentage of Test Suite Required for Complete Fault Coverage). Findings: The proposed technique m-ACO proves its efficiency on both the parameters. m-ACO achieves higher fault detection rate with minimized test suite as comparative to other meta-heuristic techniques for test case prioritization. Improvements: The proposed technique m-ACO basically works by modifying the food source searching and selection pattern of the real ants. Real ants grab every type food source it comes across; while modified ants evaluate the food fitness and uniqueness before selection. This phenomenon enhances the quality and diversity of deposited food source. Keywords: Fault Coverage, Genetic Algorithm, Regression Testing, Software Testing, Test Suite Prioritization

A Comparative Evaluation of “m-ACO” Technique for Test Suite Prioritization

Objectives: The novel test case prioritization technique “m-ACO” (“Modified Ant Colony Optimization”) for regression testing has been comparatively evaluated. Methods: “m-ACO” prioritize the test cases by altering the food source selection criteria of natural ants to enhance fault diversity. The code for the proposed technique for prioritizing test case “m-ACO” has been implemented in Perl language. This paper makes a comparative evaluation of proposed “m-ACO” technique for prioritization of test cases with GA (“Genetic Algorithm”), BCO (“Bee Colony Optimization”) Algorithms and ACO (“Ant Colony Optimization”) Algorithms using three case studies. Two metrics namely APFD (“Average Percentage of Faults Detected”) and PTR (“Percentage of Test Suite Required for Complete Fault Coverage”) have been used to measure the effectiveness of the proposed “m-ACO” technique. Findings: The proposed technique “m-ACO” produced optimal or near optimal solutions. The proposed “m-ACO” technique proves its efficiency in comparison to GA, BCO and ACO methods individually. Improvements: The proposed technique improves the ACO method by altering food source selection criteria of natural ants. The future work in this direction will comparatively evaluate the proposed “m-ACO” technique using some well known software testing problems and open source software. An automated tool for the proposed technique is being developed.

Analysing the fitness landscape of search-based software testing problems

Search-based software testing automatically derives test inputs for a software system with the goal of improving various criteria, such as branch coverage. In many cases, evolutionary algorithms are implemented to find near-optimal test suites for software systems. The result of the search is usually received without any indication of how successful the search has been. Fitness landscape characterisation can help understand the search process and its probability of success. In this study, we recorded the information content, negative slope coefficient and the number of improvements during the progress of a genetic algorithm within the EvoSuite framework. Correlating the metrics with the branch and method coverages and the fitness function values reveals that the problem formulation used in EvoSuite could be improved by revising the objective function. It also demonstrates that given the current formulation, the use of crossover has no benefits for the search as the most problematic landscape features are not the number of local optima but the presence of many plateaus.

Test Cases Generation on Robotics for basis Path Testing using Genetic Algorithm

The paper explores the Genetic Algorithm approach to generate adequate and accurate test data for a specific target path. Software plays an important role in many of the systems, where the usage of software for a variety of purposes in different domains of modern life is rapidly increasing. With advancements in technology, it becomes quite complex whereas, software often contains errors. So testing consumes more money and time, which leads to automation that reduces human effort in finding bugs and errors. Actually, Automation in the last phase of system development is similar to manual testing. In both cases, bugs are detected only after code has been complete. Software testing is the most important component of software development process. Path testing is a popular structural testing method which uses the source code of a program to find every possible executable path. Test data generation is a key problem in software testing and its automation will improve the efficiency and effectiveness of software testing. Genetic Algorithm is an adaptive heuristic search algorithm that premise on evolutionary ideas of natural selection and genetic. In this paper, Genetic Algorithm is used to generate path by converting the program into its corresponding Control Flow Graph and then automatically generates the test data for the target path using different sets of GA operators.

Automated metamorphic testing of variability analysis tools

SummaryVariability determines the capability of software applications to be configured and customized. A common need during the development of variability‐intensive systems is the automated analysis of their underlying variability models, for example, detecting contradictory configuration options. The analysis operations that are performed on variability models are often very complex, which hinders the testing of the corresponding analysis tools and makes difficult, often infeasible, to determine the correctness of their outputs, that is, the well‐known oracle problem in software testing. In this article, we present a generic approach for the automated detection of faults in variability analysis tools overcoming the oracle problem. Our work enables the generation of random variability models together with the exact set of valid configurations represented by these models. These test data are generated from scratch using stepwise transformations and assuring that certain constraints (a.k.a. metamorphic relations) hold at each step. To show the feasibility and generalizability of our approach, it has been used to automatically test several analysis tools in three variability domains: feature models, common upgradeability description format documents and Boolean formulas. Among other results, we detected 19 real bugs in 7 out of the 15 tools under test. Copyright © 2015 John Wiley &amp; Sons, Ltd.