Data-driven Optimization Problems Research Articles

On regularization schemes for data-driven optimization based on Cressie–Read divergence and CVaR

On regularization schemes for data-driven optimization based on Cressie–Read divergence and CVaR

Edge-Cloud Co-Evolutionary Algorithms for Distributed Data-Driven Optimization Problems.

Surrogate-assisted evolutionary algorithms (EAs) have been proposed in recent years to solve data-driven optimization problems. Most existing surrogate-assisted EAs are for centralized optimization and do not take into account the challenges brought by the distribution of data at the edge of networks in the era of the Internet of Things. To this end, we propose edge-cloud co-EAs (ECCoEAs) to solve distributed data-driven optimization problems, where data are collected by edge servers. Specifically, we first propose a distributed framework of ECCoEAs, which consists of a communication mechanism, edge model management, and cloud model management. This communication mechanism is to avoid deadlock during the collaboration of edge servers and the cloud server. In edge model management, the edge models are trained based on local historical data and data composed of new solutions generated by co-evolutionary and their real evaluation values. In cloud model management, the black-box prediction functions received from edge models are used to find promising solutions to guide the edge model management. Moreover, two ECCoEAs are implemented, which proves the generality of the framework. To verify the performance of algorithms for distributed data-driven optimization problems, we design a novel benchmark test suite. The performance on the benchmarks and practical distributed clustering problems shows the effectiveness of ECCoEAs.

Performance Indicator-Based Adaptive Model Selection for Offline Data-Driven Multiobjective Evolutionary Optimization.

A number of real-world multiobjective optimization problems (MOPs) are driven by the data from experiments or computational simulations. In some cases, no new data can be sampled during the optimization process and only a certain amount of data can be sampled before optimization starts. Such problems are known as offline data-driven MOPs. Although multiple surrogate models approximating each objective function are able to replace the real fitness evaluations in evolutionary algorithms (EAs), their approximation errors are easily accumulated and therefore, mislead the solution ranking. To mitigate this issue, a new surrogate-assisted indicator-based EA for solving offline data-driven multiobjective problems is proposed. The proposed algorithm adopts an indicator-based selection EA as the baseline optimizer due to its selection robustness to the approximation errors of surrogate models. Both the Kriging models and radial basis function networks (RBFNs) are employed as surrogate models. An adaptive model selection mechanism is designed to choose the right type of models according to a maximum acceptable approximation error that is less likely to mislead the indicator-based search. The main idea is that when the uncertainty of the Kriging models exceeds the acceptable error, the proposed algorithm selects RBFNs as the surrogate models. The results comparing with state-of-the-art algorithms on benchmark problems with up to ten objectives indicate that the proposed algorithm is effective on offline data-driven optimization problems with up to 20 and 30 decision variables.

Multi-condition operational optimization with adaptive knowledge transfer for municipal solid waste incineration process

To satisfy incineration efficiency and nitrogen oxides ultra-low emission requirements, it is of great importance to improve the operational performance of the municipal solid waste incineration (MSWI) process. However, due to the complex and fluctuating composition of municipal solid waste, it is challenging to guarantee optimal operation under changing conditions. To address this issue, a multi-condition operational optimization method based on adaptive knowledge transfer (MCOO-TR) is proposed for the MSWI process. First, an integrated optimization scheme, where different operation conditions are identified based on cluster analysis and expert knowledge, is developed. Second, a surrogate-assisted multi-objective particle swarm optimization algorithm is designed to solve the data-driven optimization problems for optimal operation, where the surrogate model selection mechanism is proposed to cope with multi-condition characteristics. Third, an adaptive knowledge transfer strategy, corresponding to condition identification, is designed aiming to improve optimization efficiency. Finally, the proposed MCOO-TR method is evaluated based on actual industrial data. The experimental results demonstrate that the MCOO-TR method can achieve satisfactory operational performance under changing operation conditions.

Explainable AI via learning to optimize

Indecipherable black boxes are common in machine learning (ML), but applications increasingly require explainable artificial intelligence (XAI). The core of XAI is to establish transparent and interpretable data-driven algorithms. This work provides concrete tools for XAI in situations where prior knowledge must be encoded and untrustworthy inferences flagged. We use the “learn to optimize” (L2O) methodology wherein each inference solves a data-driven optimization problem. Our L2O models are straightforward to implement, directly encode prior knowledge, and yield theoretical guarantees (e.g. satisfaction of constraints). We also propose use of interpretable certificates to verify whether model inferences are trustworthy. Numerical examples are provided in the applications of dictionary-based signal recovery, CT imaging, and arbitrage trading of cryptoassets. Code and additional documentation can be found at https://xai-l2o.research.typal.academy.

On Nonstationary Gaussian Process Model for Solving Data-Driven Optimization Problems.

In data-driven evolutionary optimization, most existing Gaussian processes (GPs)-assisted evolutionary algorithms (EAs) adopt stationary GPs (SGPs) as surrogate models, which might be insufficient for solving most optimization problems. This article finds that GPs in the optimization problems are nonstationary with great probability. We propose to employ a nonstationary GP (NSGP) surrogate model for data-driven evolutionary optimization, where the mean of the NSGP is allowed to vary with the decision variables, while its residue variance follows an SGP. In this article, the nonstationarity of GPs in the tested functions is theoretically analyzed. In addition, this article constructs an NSGP where the SGP is a degenerate case. Performance comparisons of the NSGP with the SGP and the NSGP-assisted EA (NSGP-MAEA) with the SGP-assisted EA (SGP-MAEA) are carried out on a set of benchmark problems and an antenna design problem. These comparison results demonstrate the competitiveness of the NSGP model.

Breaking the Convergence Barrier: Optimization via Fixed-Time Convergent Flows

Accelerated gradient methods are the cornerstones of large-scale, data-driven optimization problems that arise naturally in machine learning and other fields concerning data analysis. We introduce a gradient-based optimization framework for achieving acceleration, based on the recently introduced notion of fixed-time stability of dynamical systems. The method presents itself as a generalization of simple gradient-based methods suitably scaled to achieve convergence to the optimizer in a fixed-time, independent of the initialization. We achieve this by first leveraging a continuous-time framework for designing fixed-time stable dynamical systems, and later providing a consistent discretization strategy, such that the equivalent discrete-time algorithm tracks the optimizer in a practically fixed number of iterations. We also provide a theoretical analysis of the convergence behavior of the proposed gradient flows, and their robustness to additive disturbances for a range of functions obeying strong convexity, strict convexity, and possibly nonconvexity but satisfying the Polyak-Łojasiewicz inequality. We also show that the regret bound on the convergence rate is constant by virtue of the fixed-time convergence. The hyperparameters have intuitive interpretations and can be tuned to fit the requirements on the desired convergence rates. We validate the accelerated convergence properties of the proposed schemes on a range of numerical examples against the state-of-the-art optimization algorithms. Our work provides insights on developing novel optimization algorithms via discretization of continuous-time flows.

On linear optimization over Wasserstein balls

Wasserstein balls, which contain all probability measures within a pre-specified Wasserstein distance to a reference measure, have recently enjoyed wide popularity in the distributionally robust optimization and machine learning communities to formulate and solve data-driven optimization problems with rigorous statistical guarantees. In this technical note we prove that the Wasserstein ball is weakly compact under mild conditions, and we offer necessary and sufficient conditions for the existence of optimal solutions. We also characterize the sparsity of solutions if the Wasserstein ball is centred at a discrete reference measure. In comparison with the existing literature, which has proved similar results under different conditions, our proofs are self-contained and shorter, yet mathematically rigorous, and our necessary and sufficient conditions for the existence of optimal solutions are easily verifiable in practice.

Comparative empirical study on constraint handling in offline data-driven evolutionary optimization

There are some practical optimization problems that can be only optimized using historical data, which is known as offline data-driven optimization problems. Since the real function evaluations are not available in the optimization process, surrogate models must replace the real fitness evaluations to guide the search. The key issue in offline data-driven optimization is how to make full use of offline data for building robust and accurate surrogate models. However, the existing offline algorithms do not consider optimization problems with constraints. Since the predicted values of the surrogate models for constraints are inaccurate, it becomes difficult to balance the objective function and the constraints during the optimization process. In this paper, we discuss a number of commonly used constraint handling techniques and combine them with an offline data-driven evolutionary algorithm. Also, four different test functions are designed with various constraints and difficulties. The results on the test function show that the multi-objective-based constraint handling technique is more likely to obtain feasible solutions, while stochastic ranking has better quality of feasible solutions.

A surrogate-assisted optimization approach for multi-response end milling of aluminum alloy AA3105

Optimization of the end milling process is a combinatorial task due to the involvement of a large number of process variables and performance characteristics. Process-specific numerical models or mathematical functions are required for the evaluation of parametric combinations in order to improve the quality of the machined parts and machining time. This problem could be categorized as the offline data-driven optimization problem. For such problems, the surrogate or predictive models are useful, which could be employed to approximate the objective functions for the optimization algorithms. This paper presents a data-driven surrogate-assisted optimizer to model the end mill cutting of aluminum alloy on a desktop milling machine. To facilitate that, material removal rate (MRR), surface roughness (Ra), and cutting forces are considered as the functions of tool diameter, spindle speed, feed rate, and depth of cut. The principal methodology is developed using a Bayesian regularized neural network (surrogate) and a beetle antennae search algorithm (optimizer) to perform the process optimization. The relationships among the process responses are studied using Kohonen’s self-organizing map. The proposed methodology is successfully compared with three different optimization techniques and shown to outperform them with improvements of 40.98% for MRR and 10.56% for Ra. The proposed surrogate-assisted optimization method is prompt and efficient in handling the offline machining data. Finally, the validation has been done using the experimental end milling cutting carried out on aluminum alloy to measure the surface roughness, material removal rate, and cutting forces using dynamometer for the optimal cutting parameters on desktop milling center. From the estimated surface roughness value of 0.4651 μm, the optimal cutting parameters have given a maximum material removal rate of 44.027 mm3/s with less amplitude of cutting force on the workpiece. The obtained test results show that more optimal surface quality and material removal can be achieved with the optimal set of parameters.

Optimal estimation of low-rank factors via feature level data fusion of multiplex signal systems

The design of fusion engines is a subject of great importance in a variety of fields. In this paper, we focus on the problem of linear fusion at the feature level for multiple signal matrices with noises, with the features being extremal eigenvectors. When given multiple similarity matrices, the objective is to find an estimate of the latent signal eigenspace. The concentration result for the inner product of features from different matrix samples is developed, utilizing the random matrix theory. Based on of the theoretical results, we proposed an efficient algorithm, EigFuse, to solve the constrained data-driven optimization problem with different level of noises. Our method is of high efficiency by comparing it with state-of-the-art baseline approaches with multiple noise levels. Comprehensive experiments on several synthetic as well as real-life networks demonstrate our method's superior performance.

High equilibrium optimizer for global optimization

With the advent of the information age, people have higher requirements for basic algorithms. Meta-heuristic algorithms have received wide attention as a high-level strategy to study and generate fully optimized solutions to data-driven optimization problems. Using the advantage of equilibrium optimizer (EO) with better balance mode, combined with the strategy of memetic algorithm, different proportion of temperature is introduced in different stages. That is, EO and thermal exchange optimization (TEO) are fused to obtain a new highly balanced optimizer (HEO). While keeping the guiding strategy and memory mode unchanged of EO, the accuracy of optimization is greatly improved. 14 well-known benchmark functions and 7 selective algorithms were used for HEO evaluation comparison experiments. On the basis of the fitness function curve, the optimal solution and other experimental data are tested statistically. The experimental results show that the improved algorithm has high accuracy and stability, but at the cost of running a little more time. Application testing of complex engineering problems is also one of the main purposes of algorithm design. In this paper, three typical engineering design problems (three truss, welded beam and rolling bearing design) are tested and the experimental results show that this algorithm has certain competitiveness and superiority in classical engineering design.

Data-Driven Evolutionary Optimization: An Overview and Case Studies

Most evolutionary optimization algorithms assume that the evaluation of the objective and constraint functions is straightforward. In solving many real-world optimization problems, however, such objective functions may not exist. Instead, computationally expensive numerical simulations or costly physical experiments must be performed for fitness evaluations. In more extreme cases, only historical data are available for performing optimization and no new data can be generated during optimization. Solving evolutionary optimization problems driven by data collected in simulations, physical experiments, production processes, or daily life are termed data-driven evolutionary optimization. In this paper, we provide a taxonomy of different data driven evolutionary optimization problems, discuss main challenges in data-driven evolutionary optimization with respect to the nature and amount of data, and the availability of new data during optimization. Real-world application examples are given to illustrate different model management strategies for different categories of data-driven optimization problems.

A Surrogate-Assisted Multiobjective Evolutionary Algorithm for Large-Scale Task-Oriented Pattern Mining

As a branch of frequent pattern mining, the task-oriented pattern mining has received increasing attention due to its broad application scenarios. The lexicographic subset tree based algorithm and the multiobjective evolutionary algorithm are two effective approaches for finding the most frequent and complete pattern in task-oriented applications. However, both suffer from heavy computational cost since their runtime increases rapidly as the transaction dataset is scaled up. To address this issue, this paper regards the task-oriented pattern mining as a data-driven optimization problem and solves it by using a surrogate-assisted multiobjective evolutionary algorithm. Based on the framework of our previous multiobjective evolutionary algorithm for task-oriented pattern mining, the proposed algorithm estimates the objective values of most solutions using an ensemble of surrogates instead of the real objective functions, thereby highly improving the efficiency of the algorithm. Experimental results on three task-oriented applications indicate that the proposed algorithm has better efficiency than state-of-the-art algorithms.

Offline Data-Driven Evolutionary Optimization Using Selective Surrogate Ensembles

In solving many real-world optimization problems, neither mathematical functions nor numerical simulations are available for evaluating the quality of candidate solutions. Instead, surrogate models must be built based on historical data to approximate the objective functions and no new data will be available during the optimization process. Such problems are known as offline data-driven optimization problems. Since the surrogate models solely depend on the given historical data, the optimization algorithm is able to search only in a very limited decision space during offline data-driven optimization. This paper proposes a new offline data-driven evolutionary algorithm to make the full use of the offline data to guide the search. To this end, a surrogate management strategy based on ensemble learning techniques developed in machine learning is adopted, which builds a large number of surrogate models before optimization and adaptively selects a small yet diverse subset of them during the optimization to achieve the best local approximation accuracy and reduce the computational complexity. Our experimental results on the benchmark problems and a transonic airfoil design example show that the proposed algorithm is able to handle offline data-driven optimization problems with up to 100 decision variables.

Optimization in liner shipping

Seaborne trade is the lynchpin in almost every international supply chain, and about 90% of non-bulk cargo worldwide is transported by container. In this survey we give an overview of data-driven optimization problems in liner shipping. Research in liner shipping is motivated by a need for handling still more complex decision problems, based on big data sets and going across several organizational entities. Moreover, liner shipping optimization problems are pushing the limits of optimization methods, creating a new breeding ground for advanced modelling and solution methods. Starting from liner shipping network design, we consider the problem of container routing and speed optimization. Next, we consider empty container repositioning and stowage planning as well as disruption management. In addition, the problem of bunker purchasing is considered in depth. In each section we give a clear problem description, bring an overview of the existing literature, and go in depth with a specific model that somehow is essential for the problem. We conclude the survey by giving an introduction to the public benchmark instances LINER-LIB. Finally, we discuss future challenges and give directions for further research.

Data-Driven Surrogate-Assisted Multi-Objective Optimization of Complex Beneficiation Operational Process

Most optimization algorithms suppose that there exist explicit evaluation functions to evaluate the candidate solutions obtained during the optimization process. However, in actual industrial processes, it is usually very difficult to build up precise mathematical models to describe complex industrial processes due to the lack of clear mechanisms. Instead, only some noisy historical data can be used, and optimization of such problem is often known as data-driven optimization. The optimization of complex beneficiation operational process is a typical data-driven optimization problem. To solve this problem, an evolutionary algorithm assisted by Gaussian process model is proposed in this paper. To be specific, a low-order neural network model is constructed by using the data collected from mineral processing factory as real objective function, and a Gaussian process model is developed as a surrogate to reduce the number of real function evaluations. We test the new method on a series of multi-objective test instances against two other algorithms. The experimental results indicate that the proposed method has the ability to achieve significant improvement at the limited budget of real function evaluations. In addition, the proposed algorithm is also successfully applied to the optimization of complex beneficiation operational process.

Optimization in liner shipping

Seaborne trade is the lynchpin in almost every international supply chain, and about 90% of non-bulk cargo worldwide is transported by container. In this survey we give an overview of data-driven optimization problems in liner shipping. Research in liner shipping is motivated by a need for handling still more complex decision problems, based on big data sets and going across several organizational entities. Moreover, liner shipping optimization problems are pushing the limits of optimization methods, creating a new breeding ground for advanced modelling and solution methods. Starting from liner shipping network design, we consider the problem of container routing and speed optimization. Next, we consider empty container repositioning and stowage planning as well as disruption management. In addition, the problem of bunker purchasing is considered in depth. In each section we give a clear problem description, bring an overview of the existing literature, and go in depth with a specific model that somehow is essential for the problem. We conclude the survey by giving an introduction to the public benchmark instances LINER-LIB. Finally, we discuss future challenges and give directions for further research.

Data-Driven Surrogate-Assisted Multiobjective Evolutionary Optimization of a Trauma System

Most existing work on evolutionary optimization assumes that there are analytic functions for evaluating the objectives and constraints. In the real world, however, the objective or constraint values of many optimization problems can be evaluated solely based on data and solving such optimization problems is often known as data-driven optimization. In this paper, we divide data-driven optimization problems into two categories, i.e., offline and online data-driven optimization, and discuss the main challenges involved therein. An evolutionary algorithm is then presented to optimize the design of a trauma system, which is a typical offline data-driven multiobjective optimization problem, where the objectives and constraints can be evaluated using incidents only. As each single function evaluation involves a large amount of patient data, we develop a multifidelity surrogate-management strategy to reduce the computation time of the evolutionary optimization. The main idea is to adaptively tune the approximation fidelity by clustering the original data into different numbers of clusters and a regression model is constructed to estimate the required minimum fidelity. Experimental results show that the proposed algorithm is able to save up to 90% of computation time without much sacrifice of the solution quality.

階層型ニューラルネットワークの学習に対するonline/batchハイブリッド型準ニュートン法の有効性に関する研究

Various techniques based on the gradient descent method have been studied as training algorithms for neural networks. Neural network training poses data-driven optimization problems in which the objective function involves the summation of loss terms over a set of data to be modeled. For a given training data set, the gradient-based algorithm operates in one of two modes: online (stochastic) or batch. In this paper, a robust training algorithm is proposed, combining "online" mode with "batch" one. The validity of the proposed algorithm is demonstrated through computer simulations compared with the previous quasi-Newton based training methods.