Parallel Nonlinear Optimization Research Articles

Scalable Parallel Nonlinear Optimization with PyNumero and Parapint

We describe PyNumero, an open-source, object-oriented programming framework in Python that supports rapid development of performant parallel algorithms for structured nonlinear programming problems (NLP’s) using the Message Passing Interface (MPI). PyNumero provides three fundamental building blocks for developing NLP algorithms: a fast interface for calculating first and second derivatives with the AMPL Solver Library (ASL), a number of interfaces to efficient linear solvers, and block-structured vectors and matrices based on NumPy, SciPy, and MPI that support distributed parallel storage and computation. PyNumero’s design enables efficient, parallel algorithm development using high-level Python syntax while keeping expensive numerical calculations in fast, compiled implementations based on languages like C and Fortran. To demonstrate the utility of PyNumero, we also present Parapint, a Python package built on PyNumero for parallel solution of dynamic optimization problems. Parapint includes a parallel interior-point solver based on Schur-Complement decomposition. We illustrate the effectiveness of PyNumero for developing parallel algorithms with both code examples and scalability analyses for parallel matrix-vector dot products, parallel solution of structured systems of linear equations using Schur-Complement decomposition, and the parallel solution of a two-dimensional PDE optimal control problem. Our numerical results show nearly perfect scaling to more than 1,000 cores for large matrix-vector dot products and structured linear systems. Moreover, we obtain more than 354 times speedup for the optimal control example. History: Accepted by Alice Smith, EIC/Ted Ralphs, Area Editor/Software Tools. Funding: This work was funded in part by the Institute for the Design of Advanced Energy Systems (IDAES) with funding from the Office of Fossil Energy, Cross-Cutting Research, U.S. Department of Energy. This work was also funded by Sandia National Laboratories Laboratory Directed Research and Development (LDRD) program. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.1272 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2021.0285 ) at ( http://dx.doi.org/10.5281/zenodo.7192328 ).

Spline-based parallel nonlinear optimization of function sequences

Nonlinear dynamical system optimization problems exist in many scientific fields, ranging from computer vision to quantitative finance. In these problems, the underlying optimized parameters exhibit a certain degree of continuity, which can be formulated as a discrete sequence of nonlinear functions. Traditionally, such problems are either solved by ad-hoc algorithms or via independent optimization of the underlying functions. The former solutions are difficult to define and develop, requiring expertise in the field, while the latter approach does not take advantage of the inherent sequential properties of the functions. This paper presents a parallel spline-based algorithm for nonlinear optimization of function sequences, with emphasis on dataset sequences that represent dynamically evolving systems. The presented algorithm provides results that are more coherent with fewer evaluations than independent optimization of the sequence functions. We elaborate on the heuristic approach, the motivation behind using splines to model dynamical systems, and the various tiers of concurrency built into the algorithm. Furthermore, we present two distributed variants of the algorithm and compare their convergence with the serial version. The performance of the algorithm is demonstrated on benchmarks and real-world problems in audio signal decomposition, small angle X-ray scattering analysis, and video tracking of arbitrary objects.

Parallel nonlinear optimization techniques for training neural networks

In this paper, we propose the use of parallel quasi-Newton (QN) optimization techniques to improve the rate of convergence of the training process for neural networks. The parallel algorithms are developed by using the self-scaling quasi-Newton (SSQN) methods. At the beginning of each iteration, a set of parallel search directions is generated. Each of these directions is selectively chosen from a representative class of QN methods. Inexact line searches are then carried out to estimate the minimum point along each search direction. The proposed parallel algorithms are tested over a set of nine benchmark problems. Computational results show that the proposed algorithms outperform other existing methods, which are evaluated over the same set of test problems.

Parallel nonlinear optimization techniques for chemical process design problems

Parallel nonlinear optimization techniques for chemical process design problems

A view of the limitations, opportunities, and challenges in parallel nonlinear optimization

The availability and power of parallel and distributed computers is having a significant impact on how expensive problems are solved in all areas of numerical computation, and is likely to have an even larger impact in the future. This paper presents a view of how the consideration of parallelism is affecting, and is likely to affect, one important field within numerical computation, the field of nonlinear optimization. It does not attempt to survey the research that has been done in parallel nonlinear optimization. Rather it presents a set of examples, drawn mainly from our own research, that illustrate many of the limitations, opportunities, and challenges inherent in incorporating parallelism into the field of nonlinear optimization. These examples include parallel methods for unconstrained optimization problems with a small to moderate number of variables, parallel methods for large block bordered systems of nonlinear equations, and parallel methods for small-scale and large-scale global optimization problems. Our overall conclusions are mixed. For most generic optimization problems with a small to moderate number of variables, the consideration of parallelism does not appear to be leading to major algorithmic innovations. For many classes of large-scale problems, however, the consideration of parallelism appears to be creating opportunities for the development of interesting new methods that may be advantageous on parallel and sometimes even on sequential computers. In addition, a number of large-scale parallel optimization algorithms exhibit irregular, coarse-grain structure, which leads to interesting computer science challenges in areas such as dynamic scheduling and load-balancing.

Parallel nonlinear optimization techniques for chemical process design problems

Parallel processing methods with constrained nonlinear optimization were developed to show the potential to efficiently perform chemical process design calculations. A sequential optimization subroutine was developed that used a SQP algorithm and the BFGS inverse Hessian update. This subroutine had parallelism introduced at various points. Most of these methods examined ways to perform and use the parallel simultaneous function and gradient evaluations. Function evaluations are generally expensive calculations in chemical process optimization. Algorithms using a parallel finite difference Hessian (PH), Stracter's parallel variable metric (PVM) update, and Freeman's projected parallel variable metric (PPVM) update were investigated. Other algorithms investigated included Schnabel's parallel partial speculative gradient evaluation and parallel line searching. The success of a global optimization algorithm using simultaneous minimization shows potential for robust and reliable global minimization of complex multiextremal problems. The results of this work show the potential of decreasing large scale problem optimization time with parallel processing.

State-of-the-art in parallel nonlinear optimization

This survey is concerned with variants of nonlinear optimization methods designed for implementation on parallel computers. First, we consider a variety of methods for unconstrained minimization. We consider a particular type of parallelism (simultaneous function and gradient evaluations), and we concentrate on the main sources of inspiration: conjugate directions, homogeneous functions, variable-metric updates, and multi-dimensional searches. The computational process for solving small and medium-size constrained optimization problems is usually based on unconstrained optimization. This provides a straightforward opportunity for the introduction of parallelism. In the present survey, however, we focus on promising approaches for solving large, well-structured constrained problems: dualization of problems with separable objective and constraint functions, and decomposition of hierarchical problems with linking variables (typical for Bender's decomposition in the linear case). Finally, we outline the key issues in future computational studies of parallel nonlinear optimization algorithms.