Test Order Problem Research Articles

Generating Optimal Class Integration Test Orders Using Genetic Algorithms

In recent years, many intelligent optimization algorithms have been applied to the class integration and test order (CITO) problem. These algorithms also have been proved to be able to efficiently solve the problem. Here, the design of fitness function is a key task to generate the optimal solution. To better solve the class integration and test order problem, we propose a new fitness function to generate the optimal solution that achieves a balanced compromise between the different measures (objectives) such as the total number of stubs and the total stubbing complexity in this paper. We used some programs to compare and evaluate the different approaches. The experimental results show that our proposed approach is encouraging to some extent in solving the class integration and test order problem.

An Approach for the Generation of Multi-Objective Algorithms Applied to the Integration and Test Order Problem

Multi-Objective Evolutionary Algorithms (MOEAs) have been successfully applied to solve hard real software engineering problems. However, to choose and design a MOEA is considered a difficult task, since there are several parameters and components to be configured. These aspects directly impact the generated solutions and the performance of MOEAs. In this sense, this paper proposes an approach for the automatic generation of MOEAs applied to the Integration and Test Order (ITO) problem. Such a problem refers to the generation of optimal sequences of units for integration testing. The approach includes a set of parameters and components of different MOEAs, and is implemented with two design algorithms: Grammatical Evolution (GE) and Iterated Racing (irace). Evaluation results are presented, comparing the MOEAs generated by both design algorithms. Furthermore, the generated MOEAs are compared to two well-known MOEAs used in the literature to solve the ITO problem. Results show that the MOEAs generated with GE and irace perform similarly, and both outperform traditional MOEAs. The approach can reduce efforts spent to design and configure MOEAs, and serves as basis for implementing solutions to other software engineering problems.

Tuning Multi-Objective Evolutionary Algorithms on Different Sized Problem Sets

Multi-Objective Evolutionary Algorithms (MOEAs) have been applied successfully for solving real-world multi-objective problems. Their success can depend highly on the configuration of their control parameters. Different tuning methods have been proposed in order to solve this problem. Tuning can be performed on a set of problem instances in order to obtain robust control parameters. However, for real-world problems, the set of problem instances at our disposal usually are not very plentiful. This raises the question: What is a sufficient number of problems used in the tuning process to obtain robust enough parameters? To answer this question, a novel method called MOCRS-Tuning was applied on different sized problem sets for the real-world integration and test order problem. The configurations obtained by the tuning process were compared on all the used problem instances. The results show that tuning greatly improves the algorithms’ performance and that a bigger subset used for tuning does not guarantee better results. This indicates that it is possible to obtain robust control parameters with a small subset of problem instances, which also substantially reduces the time required for tuning.

A pattern-driven solution for designing multi-objective evolutionary algorithms

Multi-objective evolutionary algorithms (MOEAs) have been widely studied in the literature, which led to the development of several frameworks and techniques to implement them. Consequently, the reusability, scalability and maintainability became fundamental concerns in the development of such algorithms. To this end, the use of design patterns (DPs) can benefit, ease and improve the design of MOEAs. DPs are reusable solutions for common design problems, which can be applied to almost any context. Despite their advantages to decrease coupling, increase flexibility, and allow an easier design extension, DPs have been underexplored for MOEA design. In order to contribute to this research topic, we propose a pattern-driven solution for the design of MOEAs. The MOEA designed with our solution is compared to another MOEA designed without it. The comparison considered: the Integration and Test Order (ITO) problem and the Traveling Salesman problem (TSP). Obtained results show that the use of this DP-driven solution allows the reuse of MOEA components, without decreasing the quality, in terms of hypervolume. This means that the developer can extend the algorithms to include other components using only object-oriented mechanisms in an easier way, while maintaining the expected results.

An optimization algorithm applied to the class integration and test order problem

In recent years, many intelligent optimization algorithms are applied to the class integration and test order problem and it has been proved that they can efficiently solve this problem. In this paper, the particle swarm optimization algorithm is applied to the class integration and test order problem. First, the initial population (test orders) is generated randomly and each test order is taken as a particle. Here, we map particles to one-dimensional space; then, a particle (an integration test order) can be represented by a position in the one-dimensional space; finally, the optimal particle (integration test order) is selected by particle swarm optimization algorithm. Also, whether the precedence table of dependency relations is introduced as the constraints in evolution can impact the effect of particle swarm optimization approach is validated through experiment. The experimental results show that the particle swarm optimization algorithm is encouraging in solving class integration and test order problem.

A multi-level feedback approach for the class integration and test order problem

Abstract Class integration and test order (CITO) problem is to devise an optimal inter-class order which can minimize stubbing efforts. The existing approach for this problem, whether it is graph-based or search-based, usually wastes a significant amount of time and efforts in finding test orders, and sometimes may devise sub-optimal solutions. To overcome this limitation, we introduce a multi-level feedback approach to better solve the CITO problem. In this method, we use a multi-level feedback strategy to calculate test profit for each class, and according to test profit, propose a reward and punishment mechanism to assess the performance of class and set its test priority. Instead of breaking cycles or searching for optimum in the previous methods, our method integrates classes by their test priority, therefore significantly reduces the running time. The experiments are conducted on five benchmark programs and eight industrial programs, and the obtained results are compared with graph-based and search-based approaches. The results indicate that our approach can minimize the stubbing cost efficiently for most programs of all typical approaches compared in this work.

A multi-objective and evolutionary hyper-heuristic applied to the Integration and Test Order Problem

The field of Search-Based Software Engineering (SBSE) has widely utilized Multi-Objective Evolutionary Algorithms (MOEAs) to solve complex software engineering problems. However, the use of such algorithms can be a hard task for the software engineer, mainly due to the significant range of parameter and algorithm choices. To help in this task, the use of Hyper-heuristics is recommended. Hyper-heuristics can select or generate low-level heuristics while optimization algorithms are executed, and thus can be generically applied. Despite their benefits, we find only a few works using hyper-heuristics in the SBSE field. Considering this fact, we describe HITO, a Hyper-heuristic for the Integration and Test Order Problem, to adaptively select search operators while MOEAs are executed using one of the selection methods: Choice Function and Multi-Armed Bandit. The experimental results show that HITO can outperform the traditional MOEAs NSGA-II and MOEA/DD. HITO is also a generic algorithm, since the user does not need to select crossover and mutation operators, nor adjust their parameters.

A multi-objective optimization approach for the integration and test order problem

A common problem found during the integration testing is to determine an order to integrate and test the units. Important factors related to stubbing costs and constraints regarding to the software development context must be considered. To solve this problem, the most promising results were obtained with multi-objective algorithms, however few algorithms and contexts have been addressed by existing works. Considering such fact, this paper aims at introducing a generic approach based on multi-objective optimization to be applied in different development contexts and with distinct multi-objective algorithms. The approach is instantiated in the object and aspect-oriented contexts, and evaluated with real systems and three algorithms: NSGA-II, SPEA2 and PAES. The algorithms are compared by using different number of objectives and four quality indicators. Results point out that the characteristics of the systems, the instantiation context and the number of objectives influence on the behavior of the algorithms. Although for more complex systems, PAES reaches better results, NSGA-II is more suitable to solve the referred problem in general cases, considering all systems and indicators.

Multi-objective optimization algorithms applied to the class integration and test order problem

In the context of object-oriented software, a common problem is the determination of test orders for the integration test of classes, known as the class integration and test order (CITO) problem. The existing approaches, based on graphs, usually generate solutions that are sub-optimal, and do not consider the different factors and measures that can affect the construction of stubs. To overcome this limitation, solutions based on genetic algorithms (GA) have presented promising results. However, the determination of a cost function, which is able to generate the best solution, is not always a trivial task, mainly for complex systems. Therefore, to better represent the CITO problem, we introduce, in this paper, a multi-objective optimization approach, to generate a set of good solutions that achieve a balanced compromise between the different measures (objectives). Three different multi-objective optimization algorithms (MOA) were implemented: Pareto ant colony, multi-objective Tabu search and non-dominated sorting GA. The approach is applied to real programs and the obtained results allow comparison with the simple GA approach and evaluation of the different MOA.

Using Coupling-Based Weights for the Class Integration and Test Order Problem

During component-based and object-oriented software development, software classes exhibit relationships that complicate integration, including method calls, inheritance and aggregation. Classes are integrated and tested in specific orders, where each class is added and tested one by one to see if it integrates successfully. A difficulty arises when cyclic dependencies exist—the functionality that is used by the first class to be tested must be mimicked by creating ‘stubs’ (sometimes called ‘mock objects’), an expensive and error-prone operation. This problem is generally called the class integration and test order (CITO) problem, and solutions must fully be automated for integration and testing to proceed smoothly and efficiently. This paper describes new techniques and algorithms to solve the CITO problem. New results include improved edge weights to more precisely model the cost of stubbing, and the use of node weights, which allows more information to be used. These weights are derived from quantitative measures of couplings between the integrated and the stubbed classes. Also, a new algorithm for computing the integration and test orders is presented. The technique is compared with an existing approach and found to be cheaper, get the same results when using edge weights exclusively, and yield better results when using node weights.