Exploratory Landscape Analysis Research Articles

Optimization test function synthesis with generative adversarial networks and adaptive neuro-fuzzy systems

This paper presents an approach to synthesizing optimization test functions that couples generative adversarial networks and adaptive neuro-fuzzy systems. A generative adversarial network produces optimization landscapes from a database of known optimization test functions, and an adaptive neuro-fuzzy system performs regression on the generated landscapes to provide closed-form expressions. These expressions can be implemented as fuzzy basis function expansions. Eight databases of two-dimensional optimization landscapes reported in the literature are used to train the generative network. Exploratory landscape analysis over the generated samples reveals that the network can lead to new optimization landscapes with features of interest. In addition, fuzzy basis function expansions provide the best approximation results when compared against two symbolic regression frameworks over several selected landscapes. Examples are used to illustrate the ability of these functions to model complex surface artifacts such as plateaus. The proposed approach can be used as a mathematical collaboration tool that couples generative artificial and computational intelligence techniques to formulate high-dimensional optimization test problems from two-dimensional synthesized functions.

Opt2Vec - a continuous optimization problem representation based on the algorithm's behavior: A case study on problem classification

Characterization of the optimization problem is a crucial task in many recent optimization research topics (e.g., explainable algorithm performance assessment, and automated algorithm selection and configuration). The state-of-the-art approaches use exploratory landscape analysis to represent the optimization problem, where for each one, a set of features is extracted using a set of candidate solutions sampled by a sampling strategy over the whole decision space. This paper proposes a novel representation of continuous optimization problems by encoding the information found in the interaction between an algorithm and an optimization problem. The new problem representation is learned using the information from the states/positions in the optimization run trajectory (i.e., the candidate solutions visited by the algorithm). With the novel representation, the problem can be characterized dynamically during the optimization run, instead of using a set of candidate solutions from the whole decision space that have never been observed by the algorithm. The novel optimization problem representation is called Opt2Vec and uses an autoencoder type of neural network to encode the information found in the interaction between an optimization algorithm and optimization problem into an embedded subspace. The Opt2Vec representation efficiency is shown by enabling different optimization problems to be successfully identified using only the information obtained from the optimization run trajectory.

Improved understanding of calibration efficiency, difficulty and parameter uniqueness of conceptual rainfall runoff models using fitness landscape metrics

The ease and efficiency with which conceptual rainfall runoff (CRR) models can be calibrated, as well as issues related to the uniqueness of their parameters, has received significant attention in literature. While several studies have tried to gain a better understanding of the underlying factors affecting these issues by examining the features of model error surfaces, this has generally been done in an ad-hoc fashion using lower-dimensional representations of higher-dimensional surfaces. In this paper, it is suggested that exploratory landscape analysis (ELA) metrics can be used to quantify key features of the error surfaces of CRR models, including their roughness and flatness, as well as their degree of optima dispersion throughout the surface. This enables key error surface features of CRR models to be compared in a consistent, efficient and easily communicable fashion for models with different combinations of attributes (e.g. model structure, catchment climate conditions, error metrics, and calibration data set lengths). Results from the application of ELA metrics to the error surfaces of 420 CRR models with different combinations of the above attributes indicate that increasing model complexity results in an increase in relative error surface roughness and relative optima dispersion and that, while increasing catchment wetness increases the relative roughness of error surfaces, it also decreases optima dispersion. This suggests that for the models considered in this study, optimisation efficiency is likely to decrease with increasing model complexity and catchment wetness, while optimisation difficulty is likely to increase and parameter uniqueness likely to decrease with model complexity and catchment dryness. While implications for choice of model complexity will need further work, this study highlights the potential value of the proposed approach to understanding the calibration efficiency, difficulty and parameter uniqueness of conceptual rainfall runoff models.

Platforms as templates: Emerging datafication dynamics in digital news outlets’ datawalls

News outlets have responded to platforms disrupting audience engagement and revenue models by pursuing more stable revenue through paywall subscriptions and membership programs. However, many outlets continue to struggle, and scholars have suggested that outlets experiment with data monetization strategies by asking readers to “pay” with their data – in other words, by implementing a “datawall.” We conducted an exploratory landscape analysis of digital news websites and interviewed digital news professionals to evaluate the extent to which datawalls actually exist in the field and to investigate the roles of platformization and datafication. We found that datawalls exist on a spectrum with a complicated relationship to paywalls and that platforms play an important role in the emergence of datawalls that calls for extending the existing conceptual model of platformization. We argue that datawalls represent attempts to adopt platform logics by accumulating audience data; thus, platforms serve as templates for digital news outlets. We discuss the implications of these findings for conceptualizing platformization and datafication.

A cross-benchmark examination of feature-based algorithm selector generalization in single-objective numerical optimization

The task of selecting the best optimization algorithm for a particular problem is known as algorithm selection (AS). This involves training a model using landscape characteristics to predict algorithm performance, but a key challenge remains: making AS models generalize effectively to new, untrained benchmark suites. This study assesses AS models’ generalizability in single-objective numerical optimization across diverse benchmark suites. Using Exploratory Landscape Analysis (ELA) and transformer-based (TransOpt) features, the research investigates their individual and combined effectiveness in AS across four benchmarks: BBOB, AFFINE, RANDOM, and ZIGZAG. AS models perform differently based on benchmark suite similarities in algorithm performance distributions and single-best solvers. When suites align, these models underperform against a baseline predicting mean algorithm performance; yet, they outperform the baseline when suites differ in performance distributions and solvers. The AS models trained using the ELA landscape features are better than the models trained using the TransOpt features on the BBOB and AFFINE benchmark suites, while the opposite is true for the RANDOM benchmark suite. Ultimately, the study reveals challenges in accurately capturing algorithm performance through problem landscape features (ELA or TransOpt), impacting AS model applicability.

CNN-HT: A Two-Stage Algorithm Selection Framework.

The No Free Lunch Theorem tells us that no algorithm can beat other algorithms on all types of problems. The algorithm selection structure is proposed to select the most suitable algorithm from a set of algorithms for an unknown optimization problem. This paper introduces an innovative algorithm selection approach called the CNN-HT, which is a two-stage algorithm selection framework. In the first stage, a Convolutional Neural Network (CNN) is employed to classify problems. In the second stage, the Hypothesis Testing (HT) technique is used to suggest the best-performing algorithm based on the statistical analysis of the performance metric of algorithms that address various problem categories. The two-stage approach can adapt to different algorithm combinations without the need to retrain the entire model, and modifications can be made in the second stage only, which is an improvement of one-stage approaches. To provide a more general structure for the classification model, we adopt Exploratory Landscape Analysis (ELA) features of the problem as input and utilize feature selection techniques to reduce the redundant ones. In problem classification, the average accuracy of classifying problems using CNN is 96%, which demonstrates the advantages of CNN compared to Random Forest and Support Vector Machines. After feature selection, the accuracy increases to 98.8%, further improving the classification performance while reducing the computational cost. This demonstrates the effectiveness of the first stage of the CNN-HT method, which provides a basis for algorithm selection. In the experiments, CNN-HT shows the advantages of the second stage algorithm as well as good performance with better average rankings in different algorithm combinations compared to the individual algorithms and another algorithm combination approach.

Generating Cheap Representative Functions for Expensive Automotive Crashworthiness Optimization

Solving real-world engineering optimization problems, such as automotive crashworthiness optimization, is extremely challenging, because the problem characteristics are oftentimes not well understood. Furthermore, typical hyperparameter optimization approaches that require a large function evaluation budget are computationally hindered, if the function evaluation is expensive, e.g., requires finite element (FE) simulation runs. In this paper, we propose an approach to characterize real-world expensive black-box optimization problems using the exploratory landscape analysis (ELA). Based on these landscape characteristics, we can identify test functions that are fast-to-evaluate and representative for hyperparameter optimization purposes. Focusing on 20 problem instances from automotive crashworthiness optimization, our results reveal that these 20 crashworthiness problems exhibit landscape features different from classical optimization benchmark test suites, such as the widely-used black-box optimization benchmarking (BBOB) problem set. In fact, these 20 problem instances belong to problem classes that are distinct from the BBOB test functions based on the clustering results. Further analysis indicates that, as far as the ELA features concern, they are most similar to problem classes of tree-based test functions. By analyzing the performance of two optimization algorithms with different hyperparameters, namely the covariance matrix adaptation evolutionary strategy (CMA-ES) and Bayesian optimization (BO), we show that the tree-based test functions are indeed representative in terms of predicting the algorithm performances. Following this, such scalable and fast-to-evaluate tree-based test functions have promising potential for automated design of an optimization algorithm for specific real-world problem classes.

OPTION: OPTImization Algorithm Benchmarking ONtology

Many optimization algorithm benchmarking platforms allow users to share their experimental data to promote reproducible and reusable research. However, different platforms use different data models and formats, which drastically complicates the identification of relevant datasets, their interpretation, and their interoperability. Therefore, a semantically rich, ontology-based, machine-readable data model that can be used by different platforms is highly desirable. In this paper, we report on the development of such an ontology, which we call OPTION (OPTImization algorithm benchmarking ONtology). Our ontology provides the vocabulary needed for semantic annotation of the core entities involved in the benchmarking process, such as algorithms, problems, and evaluation measures. It also provides means for automatic data integration, improved interoperability, and powerful querying capabilities, thereby increasing the value of the benchmarking data. We demonstrate the utility of OPTION, by annotating and querying a corpus of benchmark performance data from the BBOB collection of the COCO framework and from the Yet Another Black-Box Optimization Benchmark (YABBOB) family of the Nevergrad environment. In addition, we integrate features of the BBOB functional performance landscape into the OPTION knowledge base using publicly available datasets with exploratory landscape analysis. Finally, we integrate the OPTION knowledge base into the IOHprofiler environment and provide users with the ability to perform meta-analysis of performance data.

An exploratory landscape analysis driven artificial bee colony algorithm with maximum entropic epistasis

Continuous real-valued optimization problems are existed in various engineering optimization scenarios. The artificial bee colony algorithm (ABC) which is inspired by the foraging behavior of bee colonies is an efficient optimization algorithm to address complex continuous real-valued optimization problems. In this paper, the maximum entropic epistasis (MEE) in the exploratory landscape analysis (ELA) is adopted in the ABC algorithm (MEEABC) to improve the performance of the ABC algorithm. The dimension interaction of the continuous functions computed by the MEE is introduced to guide the search process of the MEEABC algorithm during the employed bee phase and onlooker bee phase. The adaptive mutation methods and the strategy of dynamic population size reduction are implemented to increase the convergence speed and dynamically ameliorate the local exploitation capability. The solutions in the fitness landscape are automatically divided into different clusters to explore the local basin of the fitness landscape via the collaboration between MEE and adaptive mutation methods. The performance of MEEABC is tested on CEC 2017 benchmark test suite. From the experimental results, the MEEABC is superior to ABC variants and state-of-art algorithms regarded with the performance of effectiveness and robustness.

Informing Multiobjective Optimization Benchmark Construction Through Instance Space Analysis

The role of carefully constructed benchmark suites in algorithm design and testing is critical. Within the continuous multiobjective optimization domain, existing suites include the general purpose ZDT, DTLZ, and WFG suites, and more recent ones specifically designed to explore the impacts of a particular problem characteristic. However, the relationship between existing suites is not clear, and the field would benefit from a “stock-take” assessment. This article investigates the coverage of current continuous multiobjective suites using the instance space analysis (ISA) methodology. Exploratory landscape analysis is used to measure critical features of each problem suite. Thereafter, we generate a 2-D visualization of the existing problem instances by locating them in the instance space, assessing their diversity, and identifying whether there are sparse areas of value to fill with new problem instances. Our findings show that the current suites are restricted in diversity when representing the entire problem instance space. We propose and evaluate three problem construction methods: 1) problem tuning; 2) toolkit hybridization; and 3) new function injection. Problem tuning is shown to generate problems surrounding existing instances, while hybridization creates problems falling between existing suites. Furthermore, utilizing the insights afforded by ISA, we show how problem features can be identified to inform the creation of new functions which fill gaps toward the boundaries of the instance space.

Use of exploratory fitness landscape metrics to better understand the impact of model structure on the difficulty of calibrating artificial neural network models

• ELA metrics can characterize the features of the error surfaces of ANNs. • Error surfaces of simple ANNs have a more well-defined overall shape. • Error surfaces of complex ANNs are flatter with many distributed, deep local optima. • Simple ANNs can be calibrated successfully using gradient-based methods. • Hybrid approach with global and local search are useful for complex ANN calibration. Artificial Neural Network (ANN) models have been used for hydrological and water resources modelling for several decades, where their calibration (“training”) has received much attention. The aim of automated calibration processes is to identify values of the model parameters that minimize an error measure between model outputs and corresponding measured values. The degree of difficulty associated with this process is a function of the nature of the relationship between changes in values of model parameters and corresponding changes in the error measure (i.e. the features of the error surface) and the automated method used to identify the combination of model parameters that minimizes the error measure (i.e. the optimization method used to find the global optimum in the error surface). While the impact of different optimization methods has been studied extensively, how the features of the error surface change for different model structures has not. In this paper, it is demonstrated that Exploratory Landscape Analysis (ELA) metrics can be used successfully to characterize the features of the error surfaces of ANN models, thereby helping to explain the reasons for an increase or decrease in calibration difficulty, and in doing so, shedding new light on findings in existing literature. Results show that the error surfaces of single-layer multi-layer perceptrons (MLPs) with fewer hidden nodes have a more well-defined overall shape and have fewer local optima, while the error surfaces of single-layer MLPs with a larger numbers of hidden nodes are flatter and have many distributed, deep local optima. Consequently, single-layer MLPs with a smaller number of hidden nodes can be calibrated successfully using gradient-based methods, such as the back-propagation algorithm, whereas single-layer MLPs with a relatively large number of hidden nodes are best calibrated using hybrid approaches that go beyond basic gradient-based methods (for example, combination of global-based evolutionary algorithms coupled with a local search).

Characterization of constrained continuous multiobjective optimization problems: A feature space perspective

Despite the increasing interest in constrained multiobjective optimization in recent years, constrained multiobjective optimization problems (CMOPs) are still insufficiently understood and characterized. For this reason, the selection of appropriate CMOPs for benchmarking is difficult and lacks a formal background. We address this issue by extending landscape analysis to constrained multiobjective optimization. By employing four exploratory landscape analysis techniques, we propose 29 landscape features (of which 19 are novel) to characterize CMOPs. These landscape features are then used to compare eight frequently used artificial test suites against a recently proposed suite consisting of real-world problems based on physical models. The experimental results reveal that the artificial test problems fail to adequately represent some realistic characteristics, such as strong negative correlation between the objectives and the overall constraint violation. Moreover, our findings show that all the studied artificial test suites have advantages and limitations, and that no “perfect” suite exists. Additionally, the effectiveness of the proposed features at predicting algorithm performance is demonstrated for two multiobjective optimization algorithms. Benchmark designers can use the obtained results to select or generate appropriate CMOP instances based on the characteristics they want to explore.

Using regression models for characterizing and comparing black box optimization problems

Characterizing black box optimization problems is critical for improved benchmarking and subsequent automation of algorithm selection and configuration. Existing approaches in exploratory landscape analysis are based on heterogeneous features that can create difficulties in applying the analysis and validating the results. In this paper, we propose a simple yet powerful method of comparing problems using regression models. Gaussian processes are used as a flexible regression model, providing an approximate characterization of the problem landscape. We use the difference between regression models of different problem instances as a measure of distance between the problems. The goodness of fit of the regression models allows us to validate the approximation that the model makes of the problem. This can subsequently be used to determine an appropriate sample size to provide a good representation of the problem. We apply our model-based framework to explore the problem relationships in the BBOB benchmark problem set. The results demonstrate that the framework is effective at representing the problem similarities in relatively high dimensions. We also evaluate the method on a set of clustering problems where a known ranking of similarity is imposed. The results show that the framework is very effective in recovering the problem similarities.

New Benchmark Functions for Single-Objective Optimization Based on a Zigzag Pattern

Benchmarking plays a crucial role in both development of new optimization methods, and in conducting proper comparisons between already existing methods, particularly in the field of evolutionary computation. In this paper, we develop new benchmark functions for bound-constrained single-objective optimization that are based on a zigzag function. The proposed zigzag function has three parameters that control its behaviour and difficulty of the resulting problems. Utilizing the zigzag function, we introduce four new functions and conduct extensive computational experiments to evaluate their performance as benchmarks. The experiments comprise of using the newly proposed functions in 100 different parameter settings for the comparison of eight different optimization algorithms, which are a mix of canonical methods and best performing methods from the Congress on Evolutionary Computation competitions. Using the results from the computational comparison, we choose some of the parametrization of the newly proposed functions to devise an ambiguous benchmark set in which each of the problems introduces a statistically significant ranking among the algorithms, but the ranking for the entire set is ambiguous with no clear dominating relationship between the algorithms. We also conduct an exploratory landscape analysis of the newly proposed benchmark functions and compare the results with the benchmark functions used in the Black-Box-Optimization-Benchmarking suite. The results suggest that the new benchmark functions are well suited for algorithmic comparisons.

On the diversity and robustness of parameterised multi-objective test suites

The development of optimisation algorithms heavily relies on comparing performance across benchmark problem suites. In continuous unconstrained multi-objective optimisation, the most popular suites are ZDT, DTLZ and WFG. For each problem in these suites, there are construction parameters that control characteristics such as the degree of multimodality or deceptiveness. Despite encouragement from the suites’ authors to do otherwise, experiments are largely performed using only the original values of these parameters. It is important to understand the robustness of these test problems, and their potential to create a diversity of challenging problem landscapes to guide future algorithm testing and development. In this paper we propose a methodology for evaluating robustness of the benchmark test problems by strategically varying construction parameters and exploring how problem difficulty and landscape characteristics are affected. Our methodology adopts both Latin Hyper-cube Sampling and a design and analysis of experiments model to construct more diverse problem instances within the benchmark problem classes. These problem variants are evaluated for eight diverse multi-objective optimisation algorithms to contribute to our understanding of problem robustness. We measure robustness of problems indirectly in terms of impacts on algorithm performance and rankings, and directly in terms of Exploratory landscape Analysis (ELA) metrics that are used to establish problem robustness from a landscape characteristics perspective. Our results show that only eleven of the 21 benchmark problems are robust for algorithms in absolute terms, nine in relative terms, and seven which provide evidence of both types of algorithm robustness. There are also nine problems which satisfy requirements for landscape robustness. Of these, only four of the 21 benchmark problem classes are robust across all measures. These results highlight the importance of diversity in selecting benchmark problems, as the majority of the test suite problems, if only default construction parameters are considered, do not support robust conclusions to be drawn in general about how algorithms perform in the presence of various constructed characteristics intended to challenge algorithms. The existing benchmark test problems are currently insufficient for understanding algorithm performance, certainly with the popularly used default parameters, and more efforts in generating diverse problem instances would serve the research community well.

An Exploratory Landscape Analysis-Based Benchmark Suite

The choice of which objective functions, or benchmark problems, should be used to test an optimization algorithm is a crucial part of the algorithm selection framework. Benchmark suites that are often used in the literature have been shown to exhibit poor coverage of the problem space. Exploratory landscape analysis can be used to quantify characteristics of objective functions. However, exploratory landscape analysis measures are based on samples of the objective function, and there is a lack of work on the appropriate choice of sample size needed to produce reliable measures. This study presents an approach to determine the minimum sample size needed to obtain robust exploratory landscape analysis measures. Based on reliable exploratory landscape analysis measures, a self-organizing feature map is used to cluster a comprehensive set of benchmark functions. From this, a benchmark suite that has better coverage of the single-objective, boundary-constrained problem space is proposed.

Analyzing randomness effects on the reliability of exploratory landscape analysis

The inherent difficulty of solving a continuous, static, bound-constrained and single-objective black-box optimization problem depends on the characteristics of the problem’s fitness landscape and the algorithm being used. Exploratory landscape analysis (ELA) uses numerical features generated via a sampling process of the search space to describe such characteristics. Despite their success in a number of applications, these features have limitations related with the computational costs associated with generating accurate results. Consequently, only approximations are available in practice which may be unreliable, leading to systemic errors. The overarching aim of this paper is to evaluate the reliability of five well-known ELA feature sets across multiple dimensions and sample sizes. For this purpose, we propose a comprehensive experimental methodology combining exploratory and statistical validation stages, which uses resampling techniques to minimize the sampling cost, and statistical significance tests to identify strengths and weaknesses of individual features. The data resulting from the methodology is collected and made available in the LEarning and OPtimization Archive of Research Data v1.0. The results show that instances of the same function can have feature values that are significantly different; hence, non-generalizable across instances, due to the effects produced by the boundary constraints. In addition, some landscape features under evaluation are highly volatile, and strongly susceptible to changes in sample size. Finally, the results show evidence of a curse of modality, meaning that the sample size should increase with the number of local optima.

Sampling Effects on Algorithm Selection for Continuous Black-Box Optimization

In this paper, we investigate how systemic errors due to random sampling impact on automated algorithm selection for bound-constrained, single-objective, continuous black-box optimization. We construct a machine learning-based algorithm selector, which uses exploratory landscape analysis features as inputs. We test the accuracy of the recommendations experimentally using resampling techniques and the hold-one-instance-out and hold-one-problem-out validation methods. The results demonstrate that the selector remains accurate even with sampling noise, although not without trade-offs.

Characteristic feature analysis of continuous optimization problems based on Variability Map of objective function for optimization algorithm configuration

AbstractAdvanced optimization algorithms with a variety of configurable parameters become increasingly difficult to apply effectively to solving optimization problems. Appropriate algorithm configuration becomes highly relevant, still remaining a computationally expensive operation. Development of machine learning methods allows to model and predict the efficiency of different solving strategies and algorithm configurations depending on properties of optimization problem to be solved. The paper suggests the Dependency Decomposition approach to reduce computational complexity of modeling the efficiency of optimization algorithm, also considering the amount of computational resources available for optimization problem solving. The approach requires development of explicit Exploratory Landscape Analysis methods to assess a variety of significant characteristic features of optimization problems. The results of feature assessment depend on the number of sample points analyzed and their location in the design space, on top of that some of methods require additional evaluations of objective function. The paper proposes new landscape analysis methods based on given points without the need of any additional objective function evaluations. An algorithm of building a so-called Full Variability Map is suggested based on informativeness criteria formulated for groups of sample points. The paper suggests Generalized Information Content method for analysis of Full Variability Map which allows to get accurate and stable estimations of objective function features. The Sectorization method of Variability Map analysis is proposed to assess characteristic features reflecting such properties of objective function that are critical for optimization algorithm efficiency. The resulting features are invariant to the scale of objective function gradients which positively affects the generalizing ability of problems classification algorithm. The procedure of the comparative study of effectiveness of landscape analysis algorithms is introduced. The results of computational experiments indicate reliability of applying the suggested landscape analysis methods to optimization problem characterization and classification.

Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis

In benchmarking theory, creating a comprehensive and uniformly distributed set of problems is a crucial first step to designing a good benchmark. However, this step is also one of the hardest, as it can be difficult to determine how to evaluate the quality of the chosen problem set.In this article, we evaluate if the field of exploratory landscape analysis can be used to develop a generalized method of visualizing a set of arbitrary optimization functions. We present a method for visually determining the distribution of problems within a benchmark set using exploratory landscape analysis combined with clustering and t-sne visualization, and evaluate and explain the visualization this methodology produces.The proposed method is evaluated on a set of benchmark problems taken from two well known state-of-the-art real-parameter single objective optimization benchmarks: the CEC Special Sessions and Competitions on Real-Parameter Single Objective optimization, and the GECCO Black-Box Optimization Benchmark workshops.The main goal of this paper is to present an analysis of how exploratory landscape analysis can be used to visualize a benchmark problem set. We show that this method can provide a clear visualization of a benchmark problem set and shows the similarities of the problems in it by placing similar problems visually close together. We also show that the problem sets of the above benchmarks have a somewhat distinct set of problems that do not overlap.In addition, by applying feature selection approaches we show that a number of landscape features provided by state-of-the-art exploratory landscape analysis libraries are redundant and that a large amount of them are not invariant to simple transforms like scaling and shifting, at least when analyzing these two datasets.