Solution Of Large-scale Optimization Problems Research Articles

Taking the human out of decomposition-based optimization via artificial intelligence, Part II: Learning to initialize

The repeated solution of large-scale optimization problems arises frequently in process systems engineering tasks. Decomposition-based solution methods have been widely used to reduce the corresponding computational time, yet their implementation has multiple steps that are difficult to configure. We propose a machine learning approach to learn the optimal initialization of such algorithms which minimizes the computational time. Active and supervised learning is used to learn a surrogate model that predicts the computational performance for a given initialization. We apply this approach to the initialization of Generalized Benders Decomposition for the solution of mixed-integer model predictive control problems. The surrogate models are used to find the optimal initialization, which corresponds to the number of initial cuts that should be added in the master problem. The results show that the proposed approach can lead to a significant reduction in solution time, and active learning can reduce the data required for learning.

A modified nonlinear conjugate gradient algorithm for unconstrained optimization and portfolio selection problems

Conjugate gradient methods play a vital role in finding solutions of large-scale optimization problems due to their simplicity to implement, low memory requirements and as well as their convergence properties. In this paper, we propose a new conjugate gradient method that has a direction satisfying the sufficient descent property. We establish global convergence of the new method under the strong Wolfe line search conditions. Numerical results show that the new method performs better than other relevant methods in the literature. Furthermore, we use the new method to solve a portfolio selection problem.

Predicting solutions of large-scale optimization problems via machine learning: A case study in blood supply chain management

Practical constrained optimization models are often large, and solving them in a reasonable time is a challenge in many applications. Further, many industries have limited access to professional commercial optimization solvers or computational power for use in their day-to-day operational decisions. In this paper, we propose a novel approach to deal with the issue of solving large operational stochastic optimization problems (SOPs) by using machine learning models. We assume that decision makers have access to facilities to optimally solve their large-scale optimization model for some initial and limited period and for some test instances. This might be through a collaborative project with research institutes or through short-term use of high-performance computing facilities. We propose that longer term support can be provided by utilizing the solutions (i.e., the optimal value of the actionable decision variables) of the stochastic optimization model from this initial period to train a machine learning model to learn optimal operational decisions in the future. In this study, the proposed approach is employed to make decisions on transshipment of blood units in a network of hospitals. We compare the decisions learned by several machine learning models with the optimal results obtained if the hospitals had access to commercial optimization solvers and computational power, and with the hospital network’s current empirical heuristic policy. The results show that using a trained neural network model reduces the average daily cost by about 29% compared with current policy, while the exact optimal policy reduces the average daily cost by 37%. Although optimization models cannot be fully replaced by machine learning, our proposed approach while not guaranteed to be optimal can improve operational decisions when optimization models are computationally expensive and infeasible for daily operational decisions in organizations such as not-for-profit and small and medium-sized enterprises.

Contribution Based Co-Evolutionary Algorithm for Large-Scale Optimization Problems

The solution of large-scale optimization problems is the key to many decision-making processes in practice. However, it is a challenging research topic when considered both the quality of solutions and the required computational time. One of the popular approaches for these problems is to divide the problems into a number of smaller sub-problems, that are then solved separately with an exchange of some information using the cooperative co-evolution (CC) concept. However, the characteristics of sub-components could be different, and their contributions to the overall performance can also be different while solving the problem. In the CC approach, it usually applies one optimizer and allocates equal computational budget to all sub-components. In this article, a new algorithm is proposed with the use of multiple optimizers, along with a need-based allocation of computational budget for the sub-components. In the proposed algorithm, a group of optimizers cooperate in an effective way to evolve the sub-components, depending on heuristic fuzzy rules. The performance of our proposed algorithm was evaluated by solving a number of large-scale global optimization benchmark functions. The empirical results show that the proposed algorithm outperforms equal allocation CC, a single selection characteristic, a single candidate optimizer and state-of-the-art algorithms.

Randomness as source for inspiring solution search methods: Music based approaches

As the world progresses towards industrialization, engineering problems become increasingly complex and it becomes even more difficult to optimize these problems. The reason for this is the increasing complexity of variables, dimensions, space complexity, and time complexity. In order to be able to cope with such a situation, randomized intelligent optimization and search algorithms are proposed to optimize numerical benchmarking problems, multi-objective problems, and solve difficult problems including a large number of variables, dimensions, constraints, and objectives. Metaheuristic random search and optimization methods are widely used to search and find the most appropriate solutions for large-scale optimization problems in an acceptable time. They are general-purposed methods that can be efficiently applied to optimization and search problems without too much modification to accommodate a specific probing. Metaheuristic optimization algorithms are generally categorized as physics, music, sociology, biology, swarm, mathematics, plant, chemistry, sports, water, and hybrid based. Although most of the intelligent metaheuristic methods are inspired by physics and biology; concepts, activities, rules, and processes in music can be an inspiration source of new intelligent optimization and search techniques. That is why; novel and efficient music inspired intelligent optimization and search methods having effective exploitation and exploration capabilities have been proposed.In this paper, music based metaheuristic optimization algorithms were gathered and analyzed for the first time. Harmony search and its versions, melody search algorithm, and method of musical composition have been examined in detail. Furthermore, their performances have been compared within both unconstrained numerical benchmark functions and constrained problems and the obtained results from music based algorithms have been comparatively studied.

Feature Extraction of the Alzheimer’s Disease Images Using Different Optimization Algorithms

Alzheimer’s disease (AD) is a type of dementia that causes problems with memory, thinking and behavior. The symptoms of the AD are usually developed slowly and got worse over time, till reach to severe enough stage which can’t interfere with daily tasks. This paper extract the most significant features from 3D MRI AD images using different optimization algorithms. Optimization algorithms are stochastic search methods that simulate the social behavior of species or the natural biological evolution. These algorithms had been used to get near-optimum solutions for large-scale optimization problems. This paper compares the formulation and results of five recent evolutionary optimization algorithms: Particle Swarm Optimization, Bat Algorithm, Genetic Algorithm, Pattern Search, and Simulated Annealing. A brief description of each of these five algorithms had been presented. These five optimization algorithm had been applied to two proposed AD feature extraction algorithms to get near-optimum number of features that gives higher accuracy. The comparisons among the algorithms are presented in terms of number of iteration, number of features and metric parameters. The results show that the Pattern Search optimization algorithm gives higher metric parameters values with lower number of iteration and lower number of features as compared to the other optimization algorithms.

Hopfield neural networks in large-scale linear optimization problems

Hopfield neural networks and affine scaling interior point methods are combined in a hybrid approach for solving linear optimization problems. The Hopfield networks perform the early stages of the optimization procedures, providing enhanced feasible starting points for both primal and dual affine scaling interior point methods, thus facilitating the steps towards optimality. The hybrid approach is applied to a set of real world linear programming problems. The results show the potential of the integrated approach, indicating that the combination of neural networks and affine scaling interior point methods can be a good alternative to obtain solutions for large-scale optimization problems.

Efficient numerical solution of parabolic optimization problems by finite element methods

We present an approach for efficient numerical solution of optimization problems governed by parabolic partial differential equations. The main ingredients are: space-time finite element discretization, second-order optimization algorithms, and storage reduction techniques. We discuss the combination of these components for the solution of large-scale optimization problems.

Using the GA and TAO toolkits for solving large-scale optimization problems on parallel computers

Challenges in the scalable solution of large-scale optimization problems include the development of innovative algorithms and efficient tools for parallel data manipulation. This article discusses two complementary toolkits from the collection of Advanced CompuTational Software (ACTS), namely, Global Arrays (GA) for parallel data management and the Toolkit for Advanced Optimization (TAO), which have been integrated to support large-scale scientific applications of unconstrained and bound constrained minimization problems. Most likely to benefit are minimization problems arising in classical molecular dynamics, free energy simulations, and other applications where the coupling among variables requires dense data structures. TAO uses abstractions for vectors and matrices so that its optimization algorithms can easily interface to distributed data management and linear algebra capabilities implemented in the GA library. The GA/TAO interfaces are available both in the traditional library mode and as components compliant with the Common Component Architecture (CCA). We highlight the design of each toolkit, describe the interfaces between them, and demonstrate their use.

Decomposing parameters of mixture Gaussian model using genetic and maximum likelihood algorithms on dental images

We present new approaches based on Genetic Algorithms (GAs), Simulated Annealing (SA) and Expectation Maximization (EM) for determining parameters of the mixture Gaussian model. GAs are adaptive search techniques designed to search for near-optimal solutions of large-scale optimization problems with multiple local maxima. It has been shown that GAs are independent of initialization parameters and can efficiently optimize functions in large search spaces while the solution obtained by EM is a function of initial parameters. There is a relatively high likelihood of achieving sub-optimal solution, due to trapping in local maxima. In this work, we propose a combination of Genetic Algorithm with EM (Interlaced GA–EM) to improve estimation of Gaussian mixture parameters. The method uses population of mixture models, rather than a single mixture, iteratively in both GA and EM to determine Gaussian mixture parameters. To assess the performance of the proposed methods, a series of Gaussian phantoms, based on the ‘Modified Shepp–Logan’ method, were created. All proposed methods were employed to estimate the tissue parameters in each phantom and applied on Micro Computed Tomography (μCT) of dental images. The proposed method offers an accurate and stable solution for parameter estimation on Gaussian mixture models, with higher likelihood of achieving global optimal minima. Obtaining such accurate parameter estimation is a key requirement for image segmentation approach, which rely on a priori knowledge of tissue model parameters.

Computing Gradients in Large-Scale Optimization Using Automatic Differentiation

The accurate and efficient computation of gradients for partially separable functions is central to the solution of large-scale optimization problems, because these functions are ubiquitous in large-scale problems. We describe two approaches for computing gradients of partially separable functions via automatic differentiation. In our experiments we employ the ADIFOR (automatic differentiation of Fortran) tool and the SparsLinC (sparse linear combination) library. We use applications from the MINPACK-2 test problem collection to compare the numerical reliability and computational efficiency of these approaches with hand-coded derivatives and approximations based on differences of function values. Our conclusion is that automatic differentiation is the method of choice, providing code for the efficient computation of the gradient without the need for tedious hand-coding.

Impact of Partial Separability on Large-Scale Optimization

ELSO is an environment for the solution of large-scale optimization problems. With ELSO the user is required to provide only code for the evaluation of a partially separable function. ELSO exploits the partial separability structure of the function to compute the gradient efficiently using automatic differentiation. We demonstrate ELSO‘s efficiency by comparing the various options available in ELSO. Our conclusion is that the hybrid option in ELSO provides performance comparable to the hand-coded option, while having the significant advantage of not requiring a hand-coded gradient or the sparsity pattern of the partially separable function. In our test problems, which have carefully coded gradients, the computing time for the hybrid AD option is within a factor of two of the hand-coded option.

Evaluation of Large-scale Optimization Problems on Vector and Parallel Architectures

The authors examine the importance of problem formulation for the solution of large-scale optimization problems on high-performance architectures. Limited memory variable metric methods are used to illustrate performance issues. It is shown that the performance of these algorithms is drastically affected by application implementation. Model applications are drawn from the MINPACK-2 test problem collection, with numerical results from a super-scalar architecture (IBM RS6000/370), a vector architecture (CRAY-2), and a massively parallel architecture (Intel DELTA).