Parallel Global Optimization Research Articles

Tuning hyperparameters of a SVM-based water demand forecasting system through parallel global optimization

Recently, the number of machine learning based water demand forecasting solutions has been significantly increasing. Different case studies have already reported practical results proving that accurate forecasts may support optimization of operations in Water Distribution Networks (WDN). However, tuning the hyper-parameters of machine leaning algorithms is still an open problem.This paper proposes a parallel global optimization model to optimize the hyperparameters of Support Vector Machine (SVM) regression trained to provide accurate water demand forecasts in the short-time horizon (i.e. 24 h). Every SVM has the first 6 hourly water consumptions as input features and a specific hourly water demand as target to be predicted, among the remaining 18. The Mean Average Percentage Error (MAPE), computed on leave-one-out validation, is the black-box objective function optimized.Moreover, a preliminary time-series clustering has been applied in order to evaluate if this can improve the accuracy of the forecasting mechanism. Time-series clustering implies that the overall number of SVMs, whose hyperparameters are optimized through parallel global optimization, increases, with a SVM trained for each cluster identified and for each hourly water demand to be predicted, making even more critical a quick tuning of the hyperparameters.Results on the urban water demand data in Milan prove that forecasting error is significantly low and that preliminary clustering allows for further reducing error while also improving computational performances.

Adjusting process count on demand for petascale global optimization

► Dynamically increase the number of processes for a complex MPI application. ► The tight coupling of some processes makes altering their count challenging. ► Help prevent long-running, memory-intensive computations from paging/crashing. ► Can improve the performance of memory-intensive computations. ► Dynamic load-balancing mechanism implemented and shown to be effective. There are many challenges that need to be met before efficient and reliable computation at the petascale is possible. Many scientific and engineering codes running at the petascale are likely to be memory intensive, which makes thrashing a serious problem for many petascale applications. One way to overcome this challenge is to use a dynamic number of processes, so that the total amount of memory available for the computation can be increased on demand. This paper describes modifications made to the massively parallel global optimization code pVTdirect in order to allow for a dynamic number of processes. In particular, the modified version of the code monitors memory use and spawns new processes if the amount of available memory is determined to be insufficient. The primary design challenges are discussed, and performance results are presented and analyzed.

A Hybrid Algorithm for Open Vehicle Routing Optimization of Coal Mines Material

This study primarily focuses on solving a practical OVRP by applying a novel hybrid adaptive Genetic Algorithm and Tabu Search (GA-TS). The GA-TS combines the GA's parallel computing and global optimization with TS's Tabu search skill and fast local search. In the GA-TS Algorithm, individuals of population do TS local search with a certain degree of probability using natural number coding. It not only ensures global optimization, but also improves the speed of computing. The Algorithm is applied to optimize the mines material transport vehicle routing of China Zhengzhou Coal Mine and power Supply Co., Ltd. Through the applications, the optimal economical transport routing has been set up and economic costs have been greatly reduced.

Properties and numerical testing of a parallel global optimization algorithm

In the framework of multistart and local search algorithms that find the global minimum of a real function f(x), $ x \in S\subseteq R^n$ , Gaviano et alias proposed a rule for deciding, as soon as a local minimum has been found, whether to perform or not a new local minimization. That rule was designed to minimize the average local computational cost $eval_1(\cdotp)$ required to move from the current local minimum to a new one. In this paper the expression of the cost $eval_2(\cdotp)$ of the entire process of getting a global minimum is found and investigated; it is shown that $eval_2(\cdotp)$ has among its components $eval_1(\cdotp)$ and can be only monotonically increasing or decreasing; that is, it exhibits the same property of $eval_1(\cdotp)$ . Moreover, a counterexample is given that shows that the optimal values given by $eval_1(\cdotp)$ and $eval_2(\cdotp)$ might not agree. Further, computational experiments of a parallel algorithm that uses the above rule are carried out in a MatLab environment.

Asynchronous parallel hybrid optimization combining DIRECT and GSS

In this paper, we explore hybrid parallel global optimization using Dividing Rectangles (DIRECT) and asynchronous generating set search (GSS). Both DIRECT and GSS are derivative-free and so require only objective function values; this makes these methods applicable to a wide variety of science and engineering problems. DIRECT is a global search method that strategically divides the search space into ever-smaller rectangles, sampling the objective function at the centre point for each rectangle. GSS is a local search method that samples the objective function at trial points around the current best point, i.e. the point with the lowest function value. Latin hypercube sampling can be used to seed GSS with a good starting point. Using a set of global optimization test problems, we compare the parallel performance of DIRECT and GSS with hybrids that combine the two methods. Our experiments suggest that the hybrid methods are much faster than DIRECT and scale better when more processors are added. This improvement in performance is achieved without any sacrifice in the quality of the solution – the hybrid methods find the global optimum whenever DIRECT does.

Discrete Differential Evolution Strategy and its Numerical Application

Differential evolution is a simple but powerful parallel global search optimization algorithm, which has been successfully used to solve single-objective optimization problems. In this paper, a novel discrete differential evolution strategy (D2E) is proposed to enhance the ability of solving the numerical optimization problems. D2E adopts the difference information from the binary evolutionary individuals and self-adaptive parameter adjustment to effectively explore the global solution space. The results of numerical simulations demonstrate that the proposed strategy is feasible and efficient, has the better metrics performances.

A new asynchronous parallel global optimization method based on simulated annealing and differential evolution

This paper presents a new asynchronous parallel global optimization method and its application to the automated device sizing in analog integrated circuit (IC) design. The method is based on the simulated annealing algorithm (SA), but incorporates features from differential evolution (DE) to improve the sampling efficiency and avoid the problems involved with the cooling schedule selection. A simple local search procedure is also incorporated to improve the fine tuning capabilities of the method. To reduce the optimization time, the method is designed as an asynchronous master-slave parallel system that allows simultaneous evaluation of several trial solutions. Comparison with simple SA and DE on a set of well-known analytical test functions confirms the method’s efficiency. The parallel efficiency of the method is also verified by optimizing the functions with 1, 2, 4, and 8 processors. The proposed approach is also applied to several real world cases of device sizing in analog IC design. The optimization results indicate that the method is capable of finding near optimal circuits. The parallel efficiency of the method is confirmed with optimization runs on 1, 2, 4, and 8 processors.

On the impact of the migration topology on the Island Model

Parallel Global Optimization Algorithms (PGOA) provide an efficient way of dealing with hard optimization problems. One method of parallelization of GOAs that is frequently applied and commonly found in the contemporary literature is the so-called Island Model (IM). In this paper, we analyze the impact of the migration topology on the performance of a PGOA which uses the Island Model. In particular we consider parallel Differential Evolution and Simulated Annealing with Adaptive Neighborhood and draw first conclusions that emerge from the conducted experiments.

Parallel Stochastic Global Optimization Using Radial Basis Functions

We develop a parallel implementation of a stochastic radial basis function (RBF) algorithm for global optimization by Regis and Shoemaker [Regis, R. G., C. A. Shoemaker. 2007a. A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19(4) 497–509]. The proposed parallel algorithm is suitable for the global optimization of computationally expensive objective functions and does not require derivatives. Each iteration of the algorithm consists of building an RBF model to approximate the expensive function and using this model to select multiple points for simultaneous function evaluation on multiple processors. The function evaluation points are selected from a set of random candidate points according to two criteria: estimated function value based on the RBF model, and minimum distance from previously evaluated points and previously selected points within each iteration. We compare the performance of our parallel stochastic RBF algorithm against alternative parallel global optimization methods, including two multistart parallel finite-difference quasi-Newton methods, a multistart implementation of Asynchronous Parallel Pattern Search [Hough, P., T. G. Kolda, V. J. Torczon. 2001. Asynchronous parallel pattern search for nonlinear optimization. SIAM J. Sci. Comput. 23(1) 134–156], a parallel implementation of Probabilistic Global Search Lausanne [Raphael, B., I. F. C. Smith. 2003. A direct stochastic algorithm for global search. Appl. Math. Comput. 146 729–758], a parallel evolutionary algorithm, and a deterministic parallel RBF algorithm by Regis and Shoemaker [Regis, R. G., C. A. Shoemaker. 2007c. Parallel radial basis function methods for the global optimization of expensive functions. Eur. J. Oper. Res. 182(2) 514–535]. We report good results for our parallel stochastic RBF method when using one, four, or eight processors in comparison with the alternatives on 20 test problems and on 3 optimization problems involving groundwater bioremediation.

A framework for parallel large-scale global optimization

The paper describes the design and implementation of BNB-Solver, an object-oriented framework for discrete and continuous parallel global optimization. The framework supports exact branch-and-bound algorithms, heuristic methods and hybrid approaches. BNB-Solver provides a support for distributed and shared memory architectures. The implementation for distributed memory machines is based on MPI and thus can run on almost any computational cluster. In order to take advantages of multicore processors we provide a separate multi-threaded implementation for shared memory platforms. We introduce a novel collaborative scheme for combining exact and heuristic search methods that provides the support for sophisticated parallel heuristics and convenient balancing between exact and heuristic methods. In the experimental results section we discuss a nonlinear programming solver and a highly efficient knapsack solver that significantly outperforms existing parallel implementations.

Parallel global optimization of functions of several variables

On the basis of the method of nonuniform coverings, a parallel method for the global optimization of Lipschitzian functions is developed. This method is implemented in C-MPI for the global minimization of functions whose gradient satisfies the Lipschitz condition. The performance of the algorithm is demonstrated using the calculation of the structure of a protein molecule as an example.

Limitations of parallel global optimization for large-scale human movement problems

Global optimization algorithms (e.g., simulated annealing, genetic, and particle swarm) have been gaining popularity in biomechanics research, in part due to advances in parallel computing. To date, such algorithms have only been applied to small- or medium-scale optimization problems (<100 design variables). This study evaluates the applicability of a parallel particle swarm global optimization algorithm to large-scale human movement problems. The evaluation was performed using two large-scale (660 design variables) optimization problems that utilized a dynamic, 27 degree-of-freedom, full-body gait model to predict new gait motions from a nominal gait motion. Both cost functions minimized a quantity that reduced the external knee adduction torque. The first one minimized footpath errors corresponding to an increased toe out angle of 15°, while the second one minimized the knee adduction torque directly without changing the footpath. Constraints on allowable changes in trunk orientation, joint angles, joint torques, centers of pressure, and ground reactions were handled using a penalty method. For both problems, a single run with a gradient-based nonlinear least squares algorithm found a significantly better solution than did 10 runs with the global particle swarm algorithm. Due to the penalty terms, the physically realistic gradient-based solutions were located within a narrow “channel” in design space that was difficult to enter without gradient information. Researchers should exercise caution when extrapolating the performance of parallel global optimizers to human movement problems with hundreds of design variables, especially when penalty terms are included in the cost function.

Multi-population co-genetic algorithm with double chain-like agents structure for parallel global numerical optimization

For the low optimization precision and long optimization time of genetic algorithm, this paper proposed a multi-population agent co-genetic algorithm with chain-like agent structure (MPAGA). This algorithm adopted multi-population parallel searching mode, close chain-like agent structure, cycle chain-like agent structure, dynamic neighborhood competition and orthogonal crossover strategy to realize parallel optimization, and has the characteristics of high optimization precision and short optimization time. In order to verify the optimization precision of this algorithm, some popular benchmark test functions were used for comparing this algorithm and a popular agent genetic algorithm (MAGA). The experimental results show that MPAGA has higher optimization precision and shorter optimization time than MAGA.

Parallel radial basis function methods for the global optimization of expensive functions

We introduce a master–worker framework for parallel global optimization of computationally expensive functions using response surface models. In particular, we parallelize two radial basis function (RBF) methods for global optimization, namely, the RBF method by Gutmann [Gutmann, H.M., 2001a. A radial basis function method for global optimization. Journal of Global Optimization 19(3), 201–227] (Gutmann-RBF) and the RBF method by Regis and Shoemaker [Regis, R.G., Shoemaker, C.A., 2005. Constrained global optimization of expensive black box functions using radial basis functions, Journal of Global Optimization 31, 153–171] (CORS-RBF). We modify these algorithms so that they can generate multiple points for simultaneous evaluation in parallel. We compare the performance of the two parallel RBF methods with a parallel multistart derivative-based algorithm, a parallel multistart derivative-free trust-region algorithm, and a parallel evolutionary algorithm on eleven test problems and on a 6-dimensional groundwater bioremediation application. The results indicate that the two parallel RBF algorithms are generally better than the other three alternatives on most of the test problems. Moreover, the two parallel RBF algorithms have comparable performances on the test problems considered. Finally, we report good speedups for both parallel RBF algorithms when using a small number of processors.

Parallel global optimization with the particle swarm algorithm

Present day engineering optimization problems often impose large computational demands, resulting in long solution times even on a modern high-end processor. To obtain enhanced computational throughput and global search capability, we detail the coarse-grained parallelization of an increasingly popular global search method, the particle swarm optimization (PSO) algorithm. Parallel PSO performance was evaluated using two categories of optimization problems possessing multiple local minima-large-scale analytical test problems with computationally cheap function evaluations and medium-scale biomechanical system identification problems with computationally expensive function evaluations. For load-balanced analytical test problems formulated using 128 design variables, speedup was close to ideal and parallel efficiency above 95% for up to 32 nodes on a Beowulf cluster. In contrast, for load-imbalanced biomechanical system identification problems with 12 design variables, speedup plateaued and parallel efficiency decreased almost linearly with increasing number of nodes. The primary factor affecting parallel performance was the synchronization requirement of the parallel algorithm, which dictated that each iteration must wait for completion of the slowest fitness evaluation. When the analytical problems were solved using a fixed number of swarm iterations, a single population of 128 particles produced a better convergence rate than did multiple independent runs performed using sub-populations (8 runs with 16 particles, 4 runs with 32 particles, or 2 runs with 64 particles). These results suggest that (1) parallel PSO exhibits excellent parallel performance under load-balanced conditions, (2) an asynchronous implementation would be valuable for real-life problems subject to load imbalance, and (3) larger population sizes should be considered when multiple processors are available.

PANMIN: sequential and parallel global optimization procedures with a variety of options for the local search strategy

We present two sequential and one parallel global optimization codes, that belong to the stochastic class, and an interface routine that enables the use of the Merlin/MCL environment as a non-interactive local optimizer. This interface proved extremely important, since it provides flexibility, effectiveness and robustness to the local search task that is in turn employed by the global procedures. We demonstrate the use of the parallel code to a molecular conformation problem. Program summary Title of program: PANMIN Catalogue identifier: ADSU Program summary URL: http://cpc.cs.qub.ac.uk/summaries/ADSU Program obtainable from: CPC Program Library, Queen's University of Belfast, N. Ireland Computer for which the program is designed and others on which it has been tested: PANMIN is designed for UNIX machines. The parallel code runs on either shared memory architectures or on a distributed system. The code has been tested on a SUN Microsystems ENTERPRISE 450 with four CPUs, and on a 48-node cluster under Linux, with both the GNU g77 and the Portland group compilers. The parallel implementation is based on MPI and has been tested with LAM MPI and MPICH Installation: University of Ioannina, Greece Programming language used: Fortran-77 Memory required to execute with typical data: Approximately O( n 2) words, where n is the number of variables No. of bits in a word: 64 No. of processors used: 1 or many Has the code been vectorised or parallelized?: Parallelized using MPI No. of bytes in distributed program, including test data, etc.: 147163 No. of lines in distributed program, including the test data, etc.: 14366 Distribution format: gzipped tar file Nature of physical problem: A multitude of problems in science and engineering are often reduced to minimizing a function of many variables. There are instances that a local optimum does not correspond to the desired physical solution and hence the search for a better solution is required. Local optimization techniques can be trapped in any local minimum. Global Optimization is then the appropriate tool. For example, solving a non-linear system of equations via optimization, one may encounter many local minima that do not correspond to solutions, i.e. they are far from zero Method of solution: PANMIN is a suite of programs for Global Optimization that take advantage of the Merlin/MCL optimization environment [1,2]. We offer implementations of two algorithms that belong to the stochastic class and use local searches either as intermediate steps or as solution refinement Restrictions on the complexity of the problem: The only restriction is set by the available memory of the hardware configuration. The software can handle bound constrained problems. The Merlin Optimization environment must be installed. Availability of an MPI installation is necessary for executing the parallel code Typical running time: Depending on the objective function

Parallel Asynchronous Global Search and the Nested Optimization Scheme

In this paper, a parallel asynchronous information algorithm for solving multidimensional Lipschitz global optimization problems, where times for evaluating the objective function can be different from point to point, is proposed. This method uses the nested optimization scheme and a new parallel asynchronous global optimization method for solving core univariate subproblems generated by the nested scheme. The properties of the scheme related to parallel computations are investigated. Global convergence conditions for the new method and theoretical conditions of speed up, which can be reached by using asynchronous parallelization in comparison with the pure sequential case, are established. Numerical experiments comparing sequential, synchronous, and asynchronous algorithms are also reported.

An asynchronous decomposition approach to parallel global optimization

We introduce an alternating decomposition approach for solving global optimization problems on parallel computers. The method is based on decomposing moves in the state space into components which can be optimized concurrently. A number of features distinguish our approach from others which have been proposed in the literature. We report the results of applying the approach to energy minimization of Lennard-Jones clusters using simulated annealing. We find that the new approach is robust, has high parallel efficiency, and it can rapidly generate a good approximation to the molecular conformation.

A parallel simulated annealing algorithm for the design of magnetic structures

A new parallel global optimization algorithm is presented and evaluated. It is based on simulated annealing and on speculative computation. It is particularly suited for parallel architectures from some units to some tens of processors and for problems characterized by a cost function whose evaluation takes a long time, features that fit well with the design of magnetic structures. Comparative tests with respect to traditional sequential SA is presented on the design of a magnet for Magnetic Resonance Imaging. >

Global multidimensional optimization on parallel computer

A parallel algorithm for solving multiextremal multidimensional global optimization problems is proposed. The algorithm is based on reducing multidimensional problems to the one-dimensional ones by applying Peano-type space-filling curves. A new parallel scheme to construct such curves is presented. For reduced optimization problems a parallel global optimization method is constructed. Sufficient conditions of global convergence are investigated. Conditions, which guarantee considerable speedup with respect to the sequential version of the algorithm, are established. Numerical experiments executed on ALLIANT FX/80 are also presented.