Proposed Test Problems Research Articles

Deep reinforcement learning as multiobjective optimization benchmarks: Problem formulation and performance assessment

The successful deployment of Deep learning in several challenging tasks has been translated into complex control problems from different domains through Deep Reinforcement Learning (DRL). Although DRL has been extensively formulated and solved as single-objective problems, nearly all real-world RL problems often feature two or more conflicting objectives, where the goal is to obtain a high-quality and diverse set of optimal policies for different objective preferences. Consequently, the development of Multi-Objective Deep Reinforcement Learning (MODRL) algorithms has gained a lot of traction in the literature. Generally, Evolutionary Algorithms (EAs) have been demonstrated to be scalable alternatives to the classical DRL paradigms when formulated as an optimization problem. Hence it is reasonable to employ Multi-objective Evolutionary Algorithms (MOEAs) to handle MODRL tasks. However, there are several factors constraining the progress of research along this line: first, there is a lack of a general problem formulation of MODRL tasks from an optimization perspective; second, there exist several challenges in performing benchmark assessments of MOEAs for MODRL problems. To overcome these limitations: (i) we present a formulation of MODRL tasks as general multi-objective optimization problems and analyze their complex characteristics from an optimization perspective; (ii) we present an end-to-end framework, termed DRLXBench, to generate MODRL benchmark test problems for seamless running of MOEAs (iii) we propose a test suite comprising of 12 MODRL problems with different characteristics such as many-objectives, degenerated Pareto fronts, concave and convex optimization problems, etc. (iv) Finally, we present and discuss baseline results on the proposed test problems using seven representative MOEAs.

Total Tardiness Minimization in a Single-Machine with Periodical Resource Constraints

In this paper we introduce a variant of the single machine considering resource restriction per period. The objective function to be minimized is the total tardiness. We proposed an integer linear programming modeling based on a bin packing formulation. In view of the NP-hardness of the introduced variant, heuristic algorithms are required to find high-quality solutions within an admissible computation times. In this sense, we present a new hybrid matheuristic called Relax-and-Fix with Variable Fixing Search (RFVFS). This innovative solution approach combines the relax-and-fix algorithm and a strategy for the fixation of decision variables based on the concept of the variable neighborhood search metaheuristic. As statistical indicators to evaluate the solution procedures under comparison, we employ the Average Relative Deviation Index (ARDI) and the Success Rate (SR). We performed extensive computational experimentation with a testbed composed by 450 proposed test problems. Considering the results for the number of jobs, the RFVFS returned ARDI and SR values of 35.6% and 41.3%, respectively. Our proposal outperformed the best solution approach available for a closely-related problem with statistical significance.

A Novel Parametric benchmark generator for dynamic multimodal optimization

• A new parametric test suite for dynamic multimodal optimization is introduced. • An innovative time-dependent nonlinear scaling function is introduced. • The regularity of the change pattern can be controlled by the user. • The robustness of the well-known Peak Ratio indicator is improved. • Numerical results of a sample method are provided for different settings. In most existing studies on dynamic multimodal optimization (DMMO), numerical simulations have been performed using the Moving Peaks Benchmark (MPB), which is a two-decade-old test suite that cannot simulate some critical aspects of DMMO problems. This study proposes the Deterministic Distortion and Rotation Benchmark (DDRB), a method to generate deterministic DMMO test problems that can simulate more diverse types of challenges when compared to existing benchmark generators for DMMO. DDRB allows for controlling the intensity of each type of challenge independently, enabling the user to pinpoint the pros and cons of a DMMO method. DDRB first develops an existing approach for generation of static multimodal functions in which the difficulty of global optimization can be controlled. Then, it proposes a scaling function to dynamically change the relative distribution, shapes, and sizes of the basins. A deterministic technique to control the regularity of the pattern in the change is also proposed. Using these components, a parametric test suite consisting of ten test problems is developed for DMMO. Mean Robust Peak Ratio for measuring the performance of DMMO methods is formulated to overcome the sensitivity of the conventional peak ratio indicator to the predefined threshold and niche radius. Numerical results of a successful multimodal optimization method, when augmented with a simple strategy to utilize previous information, are provided on the proposed test problems in different scenarios with the aim of serving as a reference for future studies.

Проверка численной эффективности методов глобальной оптимизации

In this paper, new difficult test problems are proposed to test the numerical efficiency of global optimization methods. These are problems of unconstrained optimization with unknown solutions. The proposed test problems are inseparable and have arbitrary dimensions. The author also proposes to include the test functions by J. Nie in the list of test functions for numerical verification of the effectiveness of methods. These functions are also inseparable functions of arbitrary dimensions with unknown solutions. The proposed test problems have many local extrema. Testing the effectiveness of global optimization methods for such functions is simplified. If the method allows improving the found solutions to test problems, then it will be more effective. The existing global optimization methods are compared with the exact quadratic regularization method developed by the author. This method is compared with known software packages that implement modern methods of global optimization. These packages include several methods. The best of them use convex relaxation of the problem to obtain estimates of solutions with subsequent use of local optimization programs. But even such powerful packages have difficulties in solving the considered test problems. Some test problems, for example, with the Rana or Egg Holder function, have been solved by different methods for over 20 years. During this time, no method has allowed obtaining results that are obtained by the method of exact quadratic regularization. For almost all complex test problems with unknown solutions, this method yielded better solutions. Sometimes the advantage of this method was significant, as is the case with the Rana test function. The essence of the exact quadratic regularization method is to transform any global optimization problem to maximize the square of the Euclidean norm of a vector on a convex set. This problem is computationally much simpler. Often, with such a transformation, the multimodal problem becomes unimodal, which is easy to solve. Keywords: test problems, global optimization, unimodal problems, multimodal problems, numerical methods.

Constrained Multiobjective Optimization: Test Problem Construction and Performance Evaluations

Constrained multiobjective optimization abounds in practical applications and is gaining growing attention in the evolutionary computation community. Artificial test problems are critical to the progress in this research area. Nevertheless, many of them lack important characteristics, such as scalability and variable dependencies, which may be essential in benchmarking modern evolutionary algorithms. This article first proposes a new framework for constrained test problem construction. This framework splits a decision vector into position and distance variables and forces their optimal values to lie on a nonlinear hypersurface such that the interdependencies can be introduced among the position ones and among the distance ones individually. In this framework, two kinds of constraints are designed to introduce convergence-hardness and diversity-hardness, respectively. The first kind introduces infeasible barriers in approaching the optima, and at the same time, makes the position and distance variables interrelate with each other. The second kind restricts the feasible optimal regions such that different shapes of Pareto fronts can be obtained. Based on this framework, we construct 16 scalable and constrained test problems covering a variety of difficulties. Then, in the second part of this article, we evaluate the performance of some state of the art on the proposed test problems, showing that they are quite challenging and there is room for further enhancement of the existing algorithms. Finally, we discuss in detail the source of difficulties presented in these new problems.

A Novel Evolutionary Algorithm for Dynamic Constrained Multiobjective Optimization Problems

To promote research on dynamic constrained multiobjective optimization, we first propose a group of generic test problems with challenging characteristics, including different modes of the true Pareto front (e.g., convexity–concavity and connectedness–disconnectedness) and the changing feasible region. Subsequently, motivated by the challenges presented by dynamism and constraints, we design a dynamic constrained multiobjective optimization algorithm with a nondominated solution selection operator, a mating selection strategy, a population selection operator, a change detection method, and a change response strategy. The designed nondominated solution selection operator can obtain a nondominated population with diversity when the environment changes. The mating selection strategy and population selection operator can adaptively handle infeasible solutions. If a change is detected, the proposed change response strategy reuses some portion of the old solutions in combination with randomly generated solutions to reinitialize the population, and a steady-state update method is designed to improve the retained previous solutions. The experimental results show that the proposed test problems can be used to clearly distinguish the performance of algorithms, and that the proposed algorithm is very competitive for solving dynamic constrained multiobjective optimization problems in comparison with state-of-the-art algorithms.

A set of new multi- and many-objective test problems for continuous optimization and a comprehensive experimental evaluation

Multi- and many-objective optimization problems have wide applications in the real world, and they have received growing attention from the evolutionary computation community. To promote the algorithm development in this area, numerous studies have been devoted to designing multi- and many-objective test problems. Most of these studies focus on handling complicated Pareto fronts (PFs), and the impact of the Pareto sets (PSs) has not been well-studied, although complicated PSs are prevalent in the real world. This paper presents a set of scalable test problems according to a new principle, which considers the geometrical properties of both PF and PS. A position function with a spherical form is proposed to introduce non-linear variable dependences to the PS, so as to simulate the variable dependencies in the real-world problems. According to the proposed principle, the first m (i.e., the number of objectives) decision variables are used to form the surface of a unit hypersphere, while the rest variables are designed to optimize a certain distance function. A set of test problems are generated by the proposed principle, which are then used to investigate six representative algorithms. The experimental results indicate that the proposed test problems pose considerable difficulties to existing algorithms, calling for the need for designing new algorithms capable of handling complicated PF and PS simultaneously.

Difficulty Adjustable and Scalable Constrained Multiobjective Test Problem Toolkit.

Multiobjective evolutionary algorithms (MOEAs) have progressed significantly in recent decades, but most of them are designed to solve unconstrained multiobjective optimization problems. In fact, many real-world multiobjective problems contain a number of constraints. To promote research on constrained multiobjective optimization, we first propose a problem classification scheme with three primary types of difficulty, which reflect various types of challenges presented by real-world optimization problems, in order to characterize the constraint functions in constrained multiobjective optimization problems (CMOPs). These are feasibility-hardness, convergence-hardness, and diversity-hardness. We then develop a general toolkit to construct difficulty adjustable and scalable CMOPs (DAS-CMOPs, or DAS-CMaOPs when the number of objectives is greater than three) with three types of parameterized constraint functions developed to capture the three proposed types of difficulty. In fact, the combination of the three primary constraint functions with different parameters allows the construction of a large variety of CMOPs, with difficulty that can be defined by a triplet, with each of its parameters specifying the level of one of the types of primary difficulty. Furthermore, the number of objectives in this toolkit can be scaled beyond three. Based on this toolkit, we suggest nine difficulty adjustable and scalable CMOPs and nine CMaOPs, to be called DAS-CMOP1-9 and DAS-CMaOP1-9, respectively. To evaluate the proposed test problems, two popular CMOEAs-MOEA/D-CDP (MOEA/D with constraint dominance principle) and NSGA-II-CDP (NSGA-II with constraint dominance principle) and two popular constrained many-objective evolutionary algorithms (CMaOEAs)-C-MOEA/DD and C-NSGA-III-are used to compare performance on DAS-CMOP1-9 and DAS-CMaOP1-9 with a variety of difficulty triplets, respectively. The experimental results reveal that mechanisms in MOEA/D-CDP may be more effective in solving convergence-hard DAS-CMOPs, while mechanisms of NSGA-II-CDP may be more effective in solving DAS-CMOPs with simultaneous diversity-, feasibility-, and convergence-hardness. Mechanisms in C-NSGA-III may be more effective in solving feasibility-hard CMaOPs, while mechanisms of C-MOEA/DD may be more effective in solving CMaOPs with convergence-hardness. In addition, none of them can solve these problems efficiently, which stimulates us to continue to develop new CMOEAs and CMaOEAs to solve the suggested DAS-CMOPs and DAS-CMaOPs.

A novel scalable test problem suite for multimodal multiobjective optimization

Abstract This paper proposes a novel scalable multimodal multiobjective test problem suite. The proposed test problems have various properties, such as presence of local Pareto optimal set (PS), scalable number of PSs, nonuniformly distributed PSs, discrete Pareto front (PF), and scalable number of variables and objectives. All of the test problems proposed in this paper are continuous optimization problems. Therefore, they can be used to measure different capacities of multimodal multiobjective continuous optimization algorithms. Moreover, a landscape visualization method for multiobjective problems is proposed to show the properties of the multimodal multiobjective test problems. Based on the landscapes, the characteristics of these problems are analyzed and characterized. Furthermore, the existing multimodal multiobjective optimization algorithms and several popular multiobjective algorithms are tested and compared with the novel test problem suite. Then, a discussion on the desired properties of multimodal multiobjective optimization algorithms and future works on multimodal multiobjective optimization are presented.

Benchmark Problems and Performance Indicators for Search of Knee Points in Multiobjective Optimization

In multiobjective optimization, it is nontrivial for decision makers to articulate preferences without a priori knowledge, which is particularly true when the number of objectives becomes large. Depending on the shape of the Pareto front, optimal solutions such as knee points may be of interest. Although several multi- and many-objective optimization test suites have been proposed, little work has been reported focusing on designing multiobjective problems whose Pareto front contains complex knee regions. Likewise, few performance indicators dedicated to evaluate an algorithm's ability of accurately locating all knee points in high-dimensional objective space have been suggested. This paper proposes a set of multiobjective optimization test problems whose Pareto front consists of complex knee regions, aiming to assess the capability of evolutionary algorithms to accurately identify all knee points. Various features related to knee points have been taken into account in designing the test problems, including symmetry, differentiability, and degeneration. These features are also combined with other challenges in solving the optimization problems, such as multimodality, linkage between decision variables, nonuniformity, and scalability of the Pareto front. The proposed test problems are scalable to both decision and objective spaces. Accordingly, new performance indicators are suggested for evaluating the capability of optimization algorithms in locating the knee points. The proposed test problems, together with the performance indicators, offer a new means to develop and assess preference-based evolutionary algorithms for solving multi- and many-objective optimization problems.

Evolutionary Search with Multiple Utopian Reference Points in Decomposition‐Based Multiobjective Optimization

Decomposition‐based multiobjective evolutionary algorithms (MOEA/Ds) have become increasingly popular in recent years. In these MOEA/Ds, evolutionary search is guided by the used weight vectors in decomposition function to approximate the Pareto front (PF). Generally, the decomposition function will be constructed by the weight vectors and the reference point, which play an important role to balance convergence and diversity during the evolutionary search. However, in most existing MOEA/Ds, only one ideal point is used as the reference point for all the evolutionary search, which is harmful to search the entire PF when tackling the problems with difficult‐to‐approximate PF boundaries. To address the above problem, this paper proposes an evolutionary search method with multiple utopian reference points in MOEA/Ds. Similar to the existing MOEA/Ds, each solution is associated with one weight vector, which provides an evolutionary search direction, while the novelty of our approach is to use multiple utopian reference points, which can provide evolutionary search directions for different weight vectors. Corner solutions are used to approximate the nadir point and then multiple utopian reference points for evolutionary search can be constructed based on the ideal point and the nadir point, which are uniformly distributed on the coordinate axis or planes. The use of these utopian points can prevent solutions to gather in the same region of PF and helps to strike a good balance of exploration and exploitation in the search space. The performance of our proposed algorithm is validated on tackling 16 recently proposed test problems with difficult‐to‐approximate PF boundaries and empirically compared to eight state‐of‐the‐art multiobjective evolutionary algorithms. The experimental results demonstrate the superiority of the proposed algorithm on solving most of the test problems adopted.

A Novel Class of Test Problems for Performance Evaluation of Niching Methods

This paper proposes a novel procedure for generating parametric scalable functions with diverse properties to strengthen numerical evaluation of niching methods. It combines three simple basic functions to form a composite multimodal function, in which the function parameter controls the number of global minima. The resultant composite function may show a variety of challenges in global optimization such as high condition, correlation, and presence of many undesirable local minima, as well as those peculiar to multimodal optimization, such as nonuniformly distributed global minima with dissimilar basin sizes. Moreover, the proposed procedure results in computationally inexpensive composite functions, when compared to those generated by available methods. This allows for benchmarking long-term success of niching methods on problems with many global minima. Six parametric benchmark functions are proposed in which the function parameter controls the number of global minima. Detailed analysis on distribution, basin shapes and sizes of the global minima is provided. One of the proposed test problems involve constraints, a matter which did not receive much attention in the past. These test functions are employed to compare some of the most successful niching methods in the literature. The numerical results disclose a drastic disparity between the performance of different niching methods on the composite functions and the existing test problems, demonstrating that the proposed composite functions simulate distinct challenges. Our findings highlight importance of the core search algorithm and challenge suitability of many niching strategies in more complicated landscapes and higher dimensions.

Test Problems for Large-Scale Multiobjective and Many-Objective Optimization

The interests in multiobjective and many-objective optimization have been rapidly increasing in the evolutionary computation community. However, most studies on multiobjective and many-objective optimization are limited to small-scale problems, despite the fact that many real-world multiobjective and many-objective optimization problems may involve a large number of decision variables. As has been evident in the history of evolutionary optimization, the development of evolutionary algorithms (EAs) for solving a particular type of optimization problems has undergone a co-evolution with the development of test problems. To promote the research on large-scale multiobjective and many-objective optimization, we propose a set of generic test problems based on design principles widely used in the literature of multiobjective and many-objective optimization. In order for the test problems to be able to reflect challenges in real-world applications, we consider mixed separability between decision variables and nonuniform correlation between decision variables and objective functions. To assess the proposed test problems, six representative evolutionary multiobjective and many-objective EAs are tested on the proposed test problems. Our empirical results indicate that although the compared algorithms exhibit slightly different capabilities in dealing with the challenges in the test problems, none of them are able to efficiently solve these optimization problems, calling for the need for developing new EAs dedicated to large-scale multiobjective and many-objective optimization.

An angle based constrained many-objective evolutionary algorithm

Having successfully handled many-objective optimization problems with box constraints only by using VaEA, a vector angle based many-objective evolutionary algorithm in our precursor study, this paper extended VaEA to solve generic constrained many-objective optimization problems. The proposed algorithm (denoted by CVaEA) differs from the original one mainly in the mating selection and the environmental selection, which are made suitable in the presence of infeasible solutions. Furthermore, we suggest a set of new constrained many-objective test problems which have different ranges of function values for all the objectives. Compared with normalized problems, this set of scaled ones is more applicable to test an algorithm’s performance. This is due to the nature property of practical problems being usually far from normalization. The proposed CVaEA was compared with two latest constrained many-objective optimization methods on the proposed test problems with up to 15 objectives, and on a constrained engineering problem from practice. It was shown by the simulation results that CVaEA could find a set of well converged and properly distributed solutions, and, compared with its competitors, obtained a better balance between convergence and diversity. This, and the original VaEA paper, together demonstrate the usefulness and efficiency of vector angle based algorithms for handling both constrained and unconstrained many-objective optimization problems.

Objective Extraction for Many-Objective Optimization Problems: Algorithm and Test Problems

For many-objective optimization problems (MaOPs), in which the number of objectives is greater than three, the performance of most existing evolutionary multi-objective optimization algorithms generally deteriorates over the number of objectives. As some MaOPs may have redundant or correlated objectives, it is desirable to reduce the number of the objectives in such circumstances. However, the Pareto solution of the reduced MaOP obtained by most of the existing objective reduction methods, based on objective selection, may not be the Pareto solution of the original MaOP. In this paper, we propose an objective extraction method (OEM) for MaOPs. It formulates the reduced objective as a linear combination of the original objectives to maximize the conflict between the reduced objectives. Subsequently, the Pareto solution of the reduced MaOP obtained by the proposed algorithm is that of the original MaOP, and the proposed algorithm can thus preserve the dominance structure as much as possible. Moreover, we propose a novel framework that features both simple and complicated Pareto set shapes for many-objective test problems with an arbitrary number of essential objectives. Within this framework, we can control the importance of essential objectives. As there is no direct performance metric for the objective reduction algorithms on the benchmarks, we present a new metric that features simplicity and usability for the objective reduction algorithms. We compare the proposed OEM with three objective reduction methods, i.e., REDGA, L-PCA, and NL-MVU-PCA, on the proposed test problems and benchmark DTLZ5 with different numbers of objectives and essential objectives. Our numerical studies show the effectiveness and robustness of the proposed approach.

Dynamic differential evolution with combined variants and a repair method to solve dynamic constrained optimization problems: an empirical study

An empirical study of the algorithm dynamic differential evolution with combined variants with a repair method (DDECV $$+$$+ Repair) in the solution of dynamic constrained optimization problems is presented. Unexplored aspects of the algorithm are of particular interest in this work: (1) the role of each one of its elements, (2) its sensitivity to different change frequencies and change severities in the objective function and the constraints, (3) its ability to detect a change and recover after it, besides its diversity handling (percentage of feasible and infeasible solutions) during the search, and (4) its performance with dynamism present in different parts of the problem. Seven performance measures, eighteen recently proposed test problems and eight algorithms found in the specialized literature are considered in four experiments. The statistically validated results indicate that DDECV $$+$$+ Repair is robust to change frequency and severity variations, and that it is particularly fast to recover after a change in the environment, but highly depends on its repair method and its memory population to provide competitive results. DDECV $$+$$+ Repair shows evidence on the convenience of keeping a proportion of infeasible solutions in the population when solving dynamic constrained optimization problems. Finally, DDECV $$+$$+ Repair is highly competitive particularly when dynamism is present in both, objective function and constraints.

Obstacles and difficulties for robust benchmark problems: A novel penalty-based robust optimisation method

This paper first identifies a substantial gap in the literature of robust optimisation relating to the simplicity, low-dimensionality, lack of bias, lack of deceptiveness, and lack of multi-modality of test problems. Five obstacles and difficulties such as desired number of variables, bias, deceptiveness, multi-modality, and flatness are then proposed to design challenging robust test problems and resolve the deficiency. A standard test suit of eight robust benchmark problems is proposed along with controlling parameters that allow researchers to adjust and achieve the desired level of difficulty. After the theoretical analysis of each proposed test function, a robust particle swarm optimisation (RPSO) algorithm and a robust genetic algorithm (RGA) are employed to investigate their effectiveness experimentally. The paper also inspects the effects of the proposed controlling parameters on the difficulty of the test problems and the proposal of a novel penalty function to penalize the solutions proportional to their sensitivity to perturbations in parameters. The results demonstrate that the proposed test problems are able to benchmark the performance of robust algorithm effectively and provide different, controllable levels of difficulty. In addition, the comparative results reveal the superior performance and merits of the proposed penalty-based method in finding robust solutions. Note: The source codes of the proposed robust test functions are publicity available at www.alimirjalili.com/RO.html.

Hindrances for robust multi-objective test problems

Despite the significant number of benchmark problems for evolutionary multi-objective optimisation algorithms, there are few in the field of robust multi-objective optimisation. This paper investigates the characteristics of the existing robust multi-objective test problems and identifies the current gaps in the literature. It is observed that the majority of the current test problems suffer from simplicity, so five hindrances are introduced to resolve this issue: bias towards non-robust regions, deceptive global non-robust fronts, multiple non-robust fronts (multi-modal search space), non-improving (flat) search spaces, and different shapes for both robust and non-robust Pareto optimal fronts. A set of 12 test functions are proposed by the combination of hindrances as challenging test beds for robust multi-objective algorithms. The paper also considers the comparison of five robust multi-objective algorithms on the proposed test problems. The results show that the proposed test functions are able to provide very challenging test beds for effectively comparing robust multi-objective optimisation algorithms. Note that the source codes of the proposed test functions are publicly available at www.alimirjalili.com/RO.html.

Novel frameworks for creating robust multi-objective benchmark problems

Robust optimization deals with considering different types of uncertainties during the optimization process in order to obtain reliable solutions, a critical issue when solving real problems. Multiple objectives are another vital aspect of real problems that should be considered during optimization. In order to benchmark the performance of different meta-heuristics, test problems are essential, as the literature shows. Despite the significant number of studies in developing multi-objective test problems, there is currently neither study on the suitability of the current robust multi-objective benchmark problems, nor standard frameworks to create them. This motivates our attempts to investigate the features of the current robust test problems and propose three novel frameworks to generate various robust multi-objective test problems with alterable parameters. As case studies, Robust Multi-Objective Particle Swarm Optimization (RMOPSO), Robust Non-dominated Sorting Genetic Algorithm (RNSGA-II), Robust Multiobjective Evolutionary Algorithm Based on Decomposition (RMOEA/D), Robust Two Local Best Multi-objective Particle Swarm Optimization (R2LB-MOPSO), and Robust Decomposition-Based Multi-objective Evolutionary Algorithm with an Ensemble of Neighborhood Sizes (RENS-MOEA/D) are benchmarked on the proposed test problems. The results show that the proposed frameworks are able to generate robust multi-objective test problems with different adjustable characteristics and levels of difficulty. In addition, the results show that the test problems generated by the proposed frameworks can provide very challenging test beds for effectively benchmarking the performance of robust meta-heuristics.

Test Problem Construction for Single-Objective Bilevel Optimization

In this paper, we propose a procedure for designing controlled test problems for single-objective bilevel optimization. The construction procedure is flexible and allows its user to control the different complexities that are to be included in the test problems independently of each other. In addition to properties that control the difficulty in convergence, the procedure also allows the user to introduce difficulties caused by interaction of the two levels. As a companion to the test problem construction framework, the paper presents a standard test suite of 12 problems, which includes eight unconstrained and four constrained problems. Most of the problems are scalable in terms of variables and constraints. To provide baseline results, we have solved the proposed test problems using a nested bilevel evolutionary algorithm. The results can be used for comparison, while evaluating the performance of any other bilevel optimization algorithm. The code related to the paper may be accessed from the website http://bilevel.org .