Global Optimization Functions Research Articles

Global optimization with a new method based on the gradient direction vector without constrained derivable objective functions

Gradient descent is a popular method in optimization. This method can be used in the case of a derivable and without constraint function. But it encountered several obstacles (the choice of initial point and the type of objective function, the solution, global or local). In this work, we propose an optimization method that is used to calculate the global optimization of mono-objective optimization functions that can be derived without constraints. This method is inspired by the gradient descent method. Their principle is to use a new entity based on the gradient direction instead of gradient. Unlike the gradient method and other methods inspired by gradient descent (conjugate gradient method and Newton method) where the choice of initial point is made in a random way. In our case, the choice of initial point is fixed at the beginning and also the form of the objective function curve is not a problem. Finally, we will end by applying our method to some examples (nonlinear function, quadratic function (not-convex, convex).

Constrained global optimization of multivariate polynomials using polynomial B-spline form and B-spline consistency prune approach

In this paper, we propose basic and improved algorithms based on polynomial B-spline form for constrained global optimization of multivariate polynomial functions. The proposed algorithms are based on a branch-and-bound framework. In improved algorithm we introduce several new ingredients, such as B-spline box consistency and B-spline hull consistency algorithm to prune the search regions and make the search more efficient. The performance of the basic and improved algorithm is tested and compared on set of test problems. The results of the tests show the superiority of the improved algorithm over the basic algorithm in terms of the chosen performance metrics for 7 out-off 11 test problems. We compare optimal value of global minimum obtained using the proposed algorithms with CENSO, GloptiPoly and several state-of-the-art NLP solvers, on set of 11 test problems. The results of the tests show the superiority of the proposed algorithm and CENSO solver (open source solver for global optimization of B-spline constrained problem) in that it always captures the global minimum to the user-specified accuracy.

Global optimization of objective functions represented by ReLU networks

Neural networks can learn complex, non-convex functions, and it is challenging to guarantee their correct behavior in safety-critical contexts. Many approaches exist to find failures in networks (e.g., adversarial examples), but these cannot guarantee the absence of failures. Verification algorithms address this need and provide formal guarantees about a neural network by answering “yes or no” questions. For example, they can answer whether a violation exists within certain bounds. However, individual “yes or no" questions cannot answer qualitative questions such as “what is the largest error within these bounds”; the answers to these lie in the domain of optimization. Therefore, we propose strategies to extend existing verifiers to perform optimization and find: (i) the most extreme failure in a given input region and (ii) the minimum input perturbation required to cause a failure. A naive approach using a bisection search with an off-the-shelf verifier results in many expensive and overlapping calls to the verifier. Instead, we propose an approach that tightly integrates the optimization process into the verification procedure, achieving better runtime performance than the naive approach. We evaluate our approach implemented as an extension of Marabou, a state-of-the-art neural network verifier, and compare its performance with the bisection approach and MIPVerify, an optimization-based verifier. We observe complementary performance between our extension of Marabou and MIPVerify.

Optimal Operation Model of Drainage Works for Minimizing Waterlogging Loss in Paddy Fields

The risk of flood or waterlogging in irrigation districts has increased due to global climate change and intensive human activities. A Model of Optimal Operation of Drainage Works (MOODW) for flat irrigation district was established by incorporating the hydrological model of waterlogging process and waterlogging loss estimation, which was solved by an optimization method of genetic algorithm. The model of waterlogging process was built based on a modified Tank model and hydrodynamic model for the ditch-river system. The waterlogging loss is calculated under the condition of inconstant inundated depth by linear interpolation. The adaptive genetic algorithm with the global optimization function was selected to solve the model. With an extreme rainfall events in Gaoyou irrigation district as cases, results showed that operation time and numbers of pumps increased; thus, operating costs were 1.4 times higher than before, but the yield loss of rice decreased by 35.4% observably. Finally, the total waterlogging loss was reduced by 33.8% compared with the traditional operation of waterlogging work. The most significant improvement was found in units with high waterlogging vulnerability. The MOODW can provide the waterlogging information visually and assist the district manager in making a reasonable decision.

A New Filled Function for Non-smooth Global Optimization and Its Applications

This paper aims to discover a global optima for non-smooth optimal issues by designing a novel filled function, which, regarded as a bridge delivering one minimizer to another minimizer, contains only one parameter for easy adjustment. Theoretically, the natures of the considered filled function are conducted as well as a new algorithm is presented. Finally, certain experimental calculations are presented to illustrate the performance of the constructed algorithm.

Global optimization of multivariable functions satisfying the Vanderbei condition

We propose two algorithms for solving global optimization problems on a hyperrectangle with an objective function satisfying the Vanderbei condition (this function is also called an \(\varepsilon \)-Lipschitz continuous function). The algorithms belong to the family of non-uniform covering methods. For the algorithms we prove propositions about convergence to an \(\varepsilon \)-solution in terms of the objective function. We illustrate the performance of the algorithms using several test numerical examples with non-Lipschitz continuous objective functions.

CoolMomentum: a method for stochastic optimization by Langevin dynamics with simulated annealing

Deep learning applications require global optimization of non-convex objective functions, which have multiple local minima. The same problem is often found in physical simulations and may be resolved by the methods of Langevin dynamics with Simulated Annealing, which is a well-established approach for minimization of many-particle potentials. This analogy provides useful insights for non-convex stochastic optimization in machine learning. Here we find that integration of the discretized Langevin equation gives a coordinate updating rule equivalent to the famous Momentum optimization algorithm. As a main result, we show that a gradual decrease of the momentum coefficient from the initial value close to unity until zero is equivalent to application of Simulated Annealing or slow cooling, in physical terms. Making use of this novel approach, we propose CoolMomentum—a new stochastic optimization method. Applying Coolmomentum to optimization of Resnet-20 on Cifar-10 dataset and Efficientnet-B0 on Imagenet, we demonstrate that it is able to achieve high accuracies.

A dimensionality reduction technique for unconstrained global optimization of functions with low effective dimensionality

Abstract We investigate the unconstrained global optimization of functions with low effective dimensionality, which are constant along certain (unknown) linear subspaces. Extending the technique of random subspace embeddings in Wang et al. (2016, J. Artificial Intelligence Res., 55, 361–387), we study a generic Random Embeddings for Global Optimization (REGO) framework that is compatible with any global minimization algorithm. Instead of the original, potentially large-scale optimization problem, within REGO, a Gaussian random, low-dimensional problem with bound constraints is formulated and solved in a reduced space. We provide novel probabilistic bounds for the success of REGO in solving the original, low effective-dimensionality problem, which show its independence of the (potentially large) ambient dimension and its precise dependence on the dimensions of the effective and embedding subspaces. These results significantly improve existing theoretical analyses by providing the exact distribution of a reduced minimizer and its Euclidean norm and by the general assumptions required on the problem. We validate our theoretical findings by extensive numerical testing of REGO with three types of global optimization solvers, illustrating the improved scalability of REGO compared with the full-dimensional application of the respective solvers.

Variable-fidelity hypervolume-based expected improvement criteria for multi-objective efficient global optimization of expensive functions

Variable-fidelity surrogate-based efficient global optimization (EGO) method with the ability to adaptively select low-fidelity (LF) and high-fidelity (HF) infill point has emerged as an alternative to further save the computational cost of the single-fidelity EGO method. However, in terms of the variable-fidelity surrogate-assisted multi-objective optimization methods, existing methods rely on empirical parameters or are unable to adaptively select LF/HF sample in the optimal search process. In this paper, two variable-fidelity hypervolume-based expected improvement criteria with analytic expressions for variable-fidelity multi-objective EGO method are proposed. The first criterion relies on the concept of variable-fidelity expected improvement matrix (VFEIM) and is obtained by aggregating the VFEIM using a simplified hypervolume-based aggregation scheme. The second criterion termed as VFEMHVI is derived analytically based on a modified hypervolume definition. Both criteria can adaptively select new LF/HF samples in the iterative optimal search process to update the variable-fidelity models towards the HF Pareto front, distinguishing the proposed methods to the rests in the open literature. The constrained versions of the two criteria are also derived for problems with constraints. The effectiveness and efficiency of the proposed methods are verified and validated over analytic problems and demonstrated by two engineering problems including aerodynamic shape optimizations of the NACA0012 and RAE2822 airfoils. The results show that the VFEMHVI combined with the normalization-based strategy to define the reference point is the most efficient one over the compared methods.

Quantum Henry gas solubility optimization algorithm for global optimization

This paper proposes an improvement on the recently introduced Henry Gas Solubility Optimization (HGSO) metaheuristic algorithm that simulates Henry’s gas law (i.e., the concentration of a gas sample in a liquid solvent is proportional to the concentration of the sample in the gas phase). As an improvement, we apply quantum theory instead of the standard procedure used in the HSGO algorithm for updating solutions. The proposed algorithm is named as Quantum HGSO (QHGSO) algorithm in this paper. The suggested changes enhance the ability of HGSO to create a counterbalance between exploitation and exploration for a better investigation of the solution space. For evaluating the capability of finding the optimal solution of our proposed algorithm, a collection of forty-seven global optimization functions is solved. Moreover, three well-known engineering problems are studied to show the performance of the QHGSO algorithm in constrained optimization problems. Comparative results with other well-known metaheuristic algorithms have shown that the QHGSO algorithm outperforms others with higher computational performance.

Adaptive Multi-Level Search for Global Optimization: An Integrated Swarm Intelligence-Metamodelling Technique

Over the last decade, metaheuristic algorithms have emerged as a powerful paradigm for global optimization of multimodal functions formulated by nonlinear problems arising from various engineering subjects. However, numerical analyses of many complex engineering design problems may be performed using finite element method (FEM) or computational fluid dynamics (CFD), by which function evaluations of population-based algorithms are repetitively computed to seek a global optimum. It is noted that these simulations become computationally prohibitive for design optimization of complex structures. To efficiently and effectively address this class of problems, an adaptively integrated swarm intelligence-metamodelling (ASIM) technique enabling multi-level search and model management for the optimal solution is proposed in this paper. The developed technique comprises two steps: in the first step, a global-level exploration for near optimal solution is performed by adaptive swarm-intelligence algorithm, and in the second step, a local-level exploitation for the fine optimal solution is studied on adaptive metamodels, which are constructed by the multipoint approximation method (MAM). To demonstrate the superiority of the proposed technique over other methods, such as conventional MAM, particle swarm optimization, hybrid cuckoo search, and water cycle algorithm in terms of computational expense associated with solving complex optimization problems, one benchmark mathematical example and two real-world complex design problems are examined. In particular, the key factors responsible for the balance between exploration and exploitation are discussed as well.

A Survey on Flower pollination algorithm

Flower pollination algorithm (FPA) is a metaheuristic algorithm that proceeds its representation from flowers' proliferation role in plants. The optimal plant reproduction strategy involves the survival of the fittest as well as the optimal reproduction of plants in terms of numbers. These factors represent the fundamentals of the FPA and are optimization-oriented. Yang developed the FPA in 2012, which has since shown superiority to other metaheuristic algorithms in solving various real-world problems, such as power and energy, signal and image processing, communications, structural design, clustering and feature selection, global function optimization, computer gaming, and wireless sensor networking. Recently, many variants of FPA have been developed by modification, hybridization, and parameter-tuning to cope with the complex nature of optimization problems this paper provides a survey of FPA and its applications.

Optimization method of electric field inverse problem based on intelligent algorithm

With the expansion of China's power grid, the scale, transmission distance and load capacity of transmission projects are all growing rapidly. Meanwhile, the problem of transmission line running state detection is also attracting more and more attention. Electric field under high voltage transmission line is taken as the research object in this paper, and the principle of electric field inverse operation is analyzed, two kinds of calculation objects and methods of nondestructive examination are are described in detail. There are two kinds of intelligent algorithms in this paper in order to solve the problem of several unknown solutions in inverse electric field operation. Correspondingly, the fitness function based on voltage value and result value is established, and the inverse electric field operation algorithm with global optimization function is proposed. A calculation example of high-voltage transmission line is also used to verify the inverse problem optimization algorithm. It is respectively proved by calculation results that the effectiveness and accuracy of the optimization method based on the two kinds of intelligent algorithms.

Consensus-based optimization on hypersurfaces: Well-posedness and mean-field limit

We introduce a new stochastic differential model for global optimization of nonconvex functions on compact hypersurfaces. The model is inspired by the stochastic Kuramoto–Vicsek system and belongs to the class of Consensus-Based Optimization methods. In fact, particles move on the hypersurface driven by a drift towards an instantaneous consensus point, computed as a convex combination of the particle locations weighted by the cost function according to Laplace’s principle. The consensus point represents an approximation to a global minimizer. The dynamics is further perturbed by a random vector field to favor exploration, whose variance is a function of the distance of the particles to the consensus point. In particular, as soon as the consensus is reached, then the stochastic component vanishes. In this paper, we study the well-posedness of the model and we derive rigorously its mean-field approximation for large particle limit.

Cyber Firefly Algorithm Based on Adaptive Memory Programming for Global Optimization

Recently, two evolutionary algorithms (EAs), the glowworm swarm optimization (GSO) and the firefly algorithm (FA), have been proposed. The two algorithms were inspired by the bioluminescence process that enables the light-mediated swarming behavior for mating or foraging. From our literature survey, we are convinced with much evidence that the EAs can be more effective if appropriate responsive strategies contained in the adaptive memory programming (AMP) domain are considered in the execution. This paper contemplates this line and proposes the Cyber Firefly Algorithm (CFA), which integrates key elements of the GSO and the FA and further proliferates the advantages by featuring the AMP-responsive strategies including multiple guiding solutions, pattern search, multi-start search, swarm rebuilding, and the objective landscape analysis. The robustness of the CFA has been compared against the GSO, FA, and several state-of-the-art metaheuristic methods. The experimental result based on intensive statistical analyses showed that the CFA performs better than the other algorithms for global optimization of benchmark functions.

A partial policy iteration ADP algorithm for nonlinear neuro-optimal control with discounted total reward

This paper constructs a partial policy iteration adaptive dynamic programming (ADP) algorithm to solve the optimal control problem of nonlinear systems with discounted total reward. Compared with traditional policy iteration ADP algorithm, the approach updates the iterative control law only in a local region of the global system state space. With the benefit of this feature, the overall computational burden at each iteration for processing units can be significantly reduced. Hence, this feature enables our algorithm to be successfully executed on low-performance devices such as smartphones, smartwatches and the Internet of Things (IoT) objects. We provide the convergency analysis to show that the generated sequence of value functions is monotonically nonincreasing and can finally reach a local optimum. In addition, the corresponding local policy space is developed theoretically for the first time. Besides, when the sequence of the local system state spaces is chosen properly, we prove that the developed algorithm is capable of finding the global optimal performance index function for the nonlinear systems. Finally, we present a numerical simulation to demonstrate the effectiveness of the proposed algorithm.

A Modified Salp Swarm Algorithm Based on the Perturbation Weight for Global Optimization Problems

Metaheuristic algorithms are often applied to global function optimization problems. To overcome the poor real-time performance and low precision of the basic salp swarm algorithm, this paper introduces a novel hybrid algorithm inspired by the perturbation weight mechanism. The proposed perturbation weight salp swarm algorithm has the advantages of a broad search scope and a strong balance between exploration and exploitation and retains a relatively low computational complexity when dealing with numerous large-scale problems. A new coefficient factor is introduced to the basic salp swarm algorithm, and new update strategies for the leader position and the followers are introduced in the search phase. The new leader position updating strategy has a specific bounded scope and strong search performance, thus accelerating the iteration process. The new follower updating strategy maintains the diversity of feasible solutions while reducing the computational load. This paper describes the application of the proposed algorithm to low-dimension and variable-dimension functions. This paper also presents iteration curves, box-plot charts, and search-path graphics to verify the accuracy of the proposed algorithm. The experimental results demonstrate that the perturbation weight salp swarm algorithm offers a better search speed and search balance than the basic salp swarm algorithm in different environments.

VROHI: Visibility Recovery for Outdoor Hazy Image in Scattering Media

Additive haze model (AHM), due to its high simplicity, has a potential to increase the efficiency of the restoration procedure of images degraded by scattering media. However, AHM is designed for hazy remote sensing data and is not suitable to be used on outdoor images. In this paper, according to the low-frequency feature (LFC) of haze, AHM is modified via gamma correction technique to make it suitable for modeling outdoor images. Benefitting from the modified AHM (MAHM), a simple yet effective method called VROHI is proposed to enhance the visibility of an outdoor hazy image. In specific, a low complexity LFC extraction method is designed by utilizing characteristic of the discrete cosine transform. Subsequently, by constructing the linear function of unknown parameters and imposing the saturation prior on MAHM, the image dehazing problem can be derived into a global optimization function. Experiments reveal that the proposed VROHI is superior to the other state-of-the-art techniques in terms of both the processing efficiency and recovery quality.

Parallel Bayesian Global Optimization of Expensive Functions

Large-Scale Parallel Bayesian Optimization

A modified sine cosine algorithm for accurate global optimization of numerical functions and multiple hydropower reservoirs operation

Sine cosine algorithm (SCA) is an emerging meta-heuristic method for the complicated global optimization problems, but still suffers from the premature convergence problem due to the loss of swarm diversity. To improve the SCA performance, this paper develops a modified sine cosine algorithm coupled with three improvement strategies, where the quasi-opposition learning strategy is used to balance global exploration and local exploitation; the random weighting agent produced by multiple leader solutions is integrated into the agent’s evolution equation to improve the convergence rate; the adaptive mutation strategy is designed to increase the swarm diversity. The proposed method is compared with several famous evolutionary methods on 12 classical test functions, 24 CEC2005 composite functions and 30 CEC2017 benchmark functions. The results show that the proposed method outperforms several control methods in both solution quality and convergence rate. Then, the long-term operation optimization of multiple hydropower reservoirs in China is chosen to testify the engineering practicality of the developed method. The simulation results indicate that in different scenarios, the proposed method can produce satisfying scheduling schemes with better objective values compared with several existing evolutionary methods. Hence, a novel optimizer is provided to handle the complicated engineering optimization problem.