Methods Softw Research Articles

An investigation of stochastic trust-region based algorithms for finite-sum minimization

This work elaborates on the TRust-region-ish (TRish) algorithm, a stochastic optimization method for finite-sum minimization problems proposed by Curtis et al. in [F.E. Curtis, K. Scheinberg, and R. Shi, A stochastic trust region algorithm based on careful step normalization, INFORMS. J. Optim. 1(3) (2019), pp. 200–220; F.E. Curtis and R. Shi, A fully stochastic second-order trust region method, Optim. Methods Softw. 37(3) (2022), pp. 844–877]. A theoretical analysis that complements the results in the literature is presented, and the issue of tuning the involved hyper-parameters is investigated. Our study also focuses on a practical version of the method, which computes the stochastic gradient by means of the inner product test and the orthogonality test proposed by Bollapragada et al. in [R. Bollapragada, R. Byrd, and J. Nocedal, Adaptive sampling strategies for stochastic optimization, SIAM. J. Optim. 28(4) (2018), pp. 3312–3343]. It is shown experimentally that this implementation improves the performance of TRish and reduces its sensitivity to the choice of the hyper-parameters.

A note on the generalized Hessian of the least squares associated with systems of linear inequalities

The goal of this note is to point out an erroneous formula for the generalized Hessian of the least squares associated with a system of linear inequalities, that was given in the paper ‘A finite Newton method for classification’ by Mangasarian (Optim. Methods Softw. 17 (2002), pp. 913–929) and reproduced multiple times in other publications. We also provide sufficient contiditions for the validity of Mangasarian's formula and show that Slater's condition guarantees that some particular elements from the set defined by Mangasarian belong to the generalized Hessian of the corresponding function.

Near-optimal tensor methods for minimizing the gradient norm of convex functions and accelerated primal–dual tensor methods

Motivated, in particular, by the entropy-regularized optimal transport problem, we consider convex optimization problems with linear equality constraints, where the dual objective has Lipschitz pth order derivatives, and develop two approaches for solving such problems. The first approach is based on the minimization of the norm of the gradient in the dual problem and then the reconstruction of an approximate primal solution. Recently, Grapiglia and Nesterov [Tensor methods for finding approximate stationary points of convex functions, Optim. Methods Softw. (2020), pp. 1–34] showed lower complexity bounds for the problem of minimizing the gradient norm of the function with Lipschitz pth order derivatives. Still, the question of optimal or near-optimal methods remained open as the algorithms presented in [Grapiglia and Nesterov, Tensor methods for finding approximate stationary points of convex functions, Optim. Methods Softw. (2020), pp. 1–34] achieve suboptimal bounds only. We close this gap by proposing two near-optimal (up to logarithmic factors) methods with complexity bounds O ~ ( ε − 2 ( p + 1 ) / ( 3 p + 1 ) ) and O ~ ( ε − 2 / ( 3 p + 1 ) ) with respect to the initial objective residual and the distance between the starting point and solution, respectively. We then apply these results (having independent interest) to our primal–dual setting. As the second approach, we propose a direct accelerated primal–dual tensor method for convex problems with linear equality constraints, where the dual objective has Lipschitz pth order derivatives. For this algorithm, we prove O ~ ( ε − 1 / ( p + 1 ) ) complexity in terms of the duality gap and the residual in the constraints. We illustrate the practical performance of the proposed algorithms in experiments on logistic regression, entropy-regularized optimal transport problem, and the minimal mutual information problem.

Approximation hierarchies for copositive cone over symmetric cone and their comparison

We first provide an inner-approximation hierarchy described by a sum-of-squares (SOS) constraint for the copositive (COP) cone over a general symmetric cone. The hierarchy is a generalization of that proposed by Parrilo (Structured semidefinite programs and semialgebraic geometry methods in Robustness and optimization, Ph.D. Thesis, California Institute of Technology, Pasadena, CA, 2000) for the usual COP cone (over a nonnegative orthant). We also discuss its dual. Second, we characterize the COP cone over a symmetric cone using the usual COP cone. By replacing the usual COP cone appearing in this characterization with the inner- or outer-approximation hierarchy provided by de Klerk and Pasechnik (SIAM J Optim 12(4):875–892, https://doi.org/10.1137/S1052623401383248, 2002) or Yıldırım (Optim Methods Softw 27(1):155–173, https://doi.org/10.1080/10556788.2010.540014, 2012), we obtain an inner- or outer-approximation hierarchy described by semidefinite but not by SOS constraints for the COP matrix cone over the direct product of a nonnegative orthant and a second-order cone. We then compare them with the existing hierarchies provided by Zuluaga et al. (SIAM J Optim 16(4):1076–1091, https://doi.org/10.1137/03060151X, 2006) and Lasserre (Math Program 144:265–276, https://doi.org/10.1007/s10107-013-0632-5, 2014). Theoretical and numerical examinations imply that we can numerically increase a depth parameter, which determines an approximation accuracy, in the approximation hierarchies derived from de Klerk and Pasechnik (SIAM J Optim 12(4):875–892, https://doi.org/10.1137/S1052623401383248, 2002) and Yıldırım (Optim Methods Softw 27(1):155–173, https://doi.org/10.1080/10556788.2010.540014, 2012), particularly when the nonnegative orthant is small. In such a case, the approximation hierarchy derived from Yıldırım (Optim Methods Softw 27(1):155–173, https://doi.org/10.1080/10556788.2010.540014, 2012) can yield nearly optimal values numerically. Combining the proposed approximation hierarchies with existing ones, we can evaluate the optimal value of COP programming problems more accurately and efficiently.

A new extension of Chubanov’s method to symmetric cones

We propose a new variant of Chubanov’s method for solving the feasibility problem over the symmetric cone by extending Roos’s method (Optim Methods Softw 33(1):26–44, 2018) of solving the feasibility problem over the nonnegative orthant. The proposed method considers a feasibility problem associated with a norm induced by the maximum eigenvalue of an element and uses a rescaling focusing on the upper bound for the sum of eigenvalues of any feasible solution to the problem. Its computational bound is (1) equivalent to that of Roos’s original method (2018) and superior to that of Lourenço et al.’s method (Math Program 173(1–2):117–149, 2019) when the symmetric cone is the nonnegative orthant, (2) superior to that of Lourenço et al.’s method (2019) when the symmetric cone is a Cartesian product of second-order cones, (3) equivalent to that of Lourenço et al.’s method (2019) when the symmetric cone is the simple positive semidefinite cone, and (4) superior to that of Pena and Soheili’s method (Math Program 166(1–2):87–111, 2017) for any simple symmetric cones under the feasibility assumption of the problem imposed in Pena and Soheili’s method (2017). We also conduct numerical experiments that compare the performance of our method with existing methods by generating strongly (but ill-conditioned) feasible instances. For any of these instances, the proposed method is rather more efficient than the existing methods in terms of accuracy and execution time.

A subgradient proximal method for solving a class of monotone multivalued variational inequality problems

It is well known that the algorithms with using a proximal operator can be not convergent for monotone variational inequality problems in the general case. Malitsky (Optim. Methods Softw. 33 (1) 140–164, ??) proposed a proximal extrapolated gradient algorithm ensuring convergence for the problems, where the constraints are a finite-dimensional vector space. Based on this proximal extrapolated gradient techniques, we propose a new subgradient proximal iteration method for solving monotone multivalued variational inequality problems with the closed convex constraint. At each iteration, two strongly convex subprograms are required to solve separately by using proximal operators. Then, the algorithm is convergent for monotone and Lipschitz continuous cost mapping. We also use the proposed algorithm to solve a jointly constrained Cournot-Nash equilibirum model. Some numerical experiment and comparison results for convex nonlinear programming confirm efficiency of the proposed modification.

An accelerated viscosity forward-backward splitting algorithm with the linesearch process for convex minimization problems

In this paper, we consider and investigate a convex minimization problem of the sum of two convex functions in a Hilbert space. The forward-backward splitting algorithm is one of the popular optimization methods for approximating a minimizer of the function; however, the stepsize of this algorithm depends on the Lipschitz constant of the gradient of the function, which is not an easy work to find in general practice. By using a new modification of the linesearches of Cruz and Nghia [Optim. Methods Softw. 31:1209–1238, 2016] and Kankam et al. [Math. Methods Appl. Sci. 42:1352–1362, 2019] and an inertial technique, we introduce an accelerated viscosity-type algorithm without any Lipschitz continuity assumption on the gradient. A strong convergence result of the proposed algorithm is established under some control conditions. As applications, we apply our algorithm to solving image and signal recovery problems. Numerical experiments show that our method has a higher efficiency than the well-known methods in the literature.

A new primal-dual interior-point method for semidefinite optimization based on a parameterized kernel function

As indicated in the recent studies about primal-dual interior-point methods (IPMs) based on kernel functions, a kernel function not only serves to determine the search direction and measure the distance of the current iteration point to the $$\mu $$ -center, but also affects the iteration complexity and the practical computational efficiency of the algorithm. In this paper, we propose a new IPM for semidefinite optimization (SDO) based on a parameterized kernel function which is a generalization of the one presented by Bai et al. (Optim Methods Softw 17(6):985–1008, 2002). By using the good properties of the parameterized kernel function, we deduce that the iteration bound for large-update method is $$O(\sqrt{n}\log {n}\log {\frac{n}{\epsilon }})$$ for $$q=O(n)$$ , which is the best known complexity results for such methods. In our knowledge, this result is the first instance of primal-dual interior point method for SDO which involving the kernel function. Some numerical results have been provided.

Secant update version of quasi-Newton PSB with weighted multisecant equations

Quasi-Newton methods are often used in the frame of non-linear optimization. In those methods, the quality and cost of the estimate of the Hessian matrix has a major influence on the efficiency of the optimization algorithm, which has a huge impact for computationally costly problems. One strategy to create a more accurate estimate of the Hessian consists in maximizing the use of available information during this computation. This is done by combining different characteristics. The Powell-Symmetric-Broyden method (PSB) imposes, for example, the satisfaction of the last secant equation, which is called secant update property, and the symmetry of the Hessian (Powell in Nonlinear Programming 31–65, 1970). Imposing the satisfaction of more secant equations should be the next step to include more information into the Hessian. However, Schnabel proved that this is impossible (Schnabel in quasi-Newton methods using multiple secant equations, 1983). Penalized PSB (pPSB), works around the impossibility by giving a symmetric Hessian and penalizing the non-satisfaction of the multiple secant equations by using weight factors (Gratton et al. in Optim Methods Softw 30(4):748–755, 2015). Doing so, he loses the secant update property. In this paper, we combine the properties of PSB and pPSB by adding to pPSB the secant update property. This gives us the secant update penalized PSB (SUpPSB). This new formula that we propose also avoids matrix inversions, which makes it easier to compute. Next to that, SUpPSB also performs globally better compared to pPSB.

Inner approximating the completely positive cone via the cone of scaled diagonally dominant matrices

Motivated by the expressive power of completely positive programming to encode hard optimization problems, many approximation schemes for the completely positive cone have been proposed and successfully used. Most schemes are based on outer approximations, with the only inner approximations available being a linear programming based method proposed by Bundfuss and Dur (SIAM J Optim 20(1):30–53, 2009) and also Yildirim (Optim Methods Softw 27(1):155–173, 2012), and a semidefinite programming based method proposed by Lasserre (Math Program 144(1):265–276, 2014). In this paper, we propose the use of the cone of nonnegative scaled diagonally dominant matrices as a natural inner approximation to the completely positive cone. Using projections of this cone we derive new graph-based second-order cone approximation schemes for completely positive programming, leading to both uniform and problem-dependent hierarchies. This offers a compromise between the expressive power of semidefinite programming and the speed of linear programming based approaches. Numerical results on random problems, standard quadratic programs and the stable set problem are presented to illustrate the effectiveness of our approach.

A generic framework for approximation analysis of greedy algorithms for star bicoloring

This paper presents a generic framework for the design and comparison of polynomial-time approximation algorithms for MINIMUM STAR BICOLORING. This generic framework is parameterized by algorithms which produce sequences of distance-2 independent sets. As our main technical result we show that, when the parameterized algorithm produces sequences of distance-2 independent sets that remove at least edges during each step, the generic framework produces a polynomial-time approximation algorithm for MINIMUM STAR BICOLORING that is always at least of optimal. Under the generic framework, we model two algorithms for MINIMUM STAR BICOLORING from the literature: Complete Direct Cover (CDC) [Hossain and Steihaug, Computing a sparse Jacobian matrix by rows and columns, Optim. Methods Softw. 10 (1998), pp. 33–48] and ASBC [Juedes and Jones, Coloring Jacobians revisited: A new algorithm for star and acyclic bicoloring, Optim. Methods Softw. 27(1–3) (2012), pp. 295–309]. We apply our main result to show approximation upper bounds of and , respectively, for these two algorithms. Our approximation upper bound for CDC is the first known approximation analysis for this algorithm. In addition to modelling CDC and ASBC, we use the generic framework to build and analyze three new approximation algorithms for MINIMUM STAR BICOLORING: MAX-NEIGHBORHOOD, MAX-RATIO , and LOCAL-SEARCH-k .

Modified inexact Levenberg–Marquardt methods for solving nonlinear least squares problems

In the present paper, we propose a modified inexact Levenberg–Marquardt method (LMM) and its global version by virtue of Armijo, Wolfe or Goldstein line-search schemes to solve nonlinear least squares problems (NLSP), especially for the underdetermined case. Under a local error bound condition, we show that a sequence generated by the modified inexact LMM converges to a solution superlinearly and even quadratically for some special parameters, which improves the corresponding results of the classical inexact LMM in Dan et al. (Optim Methods Softw 17:605–626, 2002). Furthermore, the quadratical convergence of the global version of the modified inexact LMM is also established. Finally, preliminary numerical experiments on some medium/large scale underdetermined NLSP show that our proposed algorithm outperforms the classical inexact LMM.

An Augmented Lagrangian Method for Equality Constrained Optimization with Rapid Infeasibility Detection Capabilities

We present a primal-dual augmented Lagrangian method for solving an equality constrained minimization problem, which is able to rapidly detect infeasibility. The method is based on a modification of the algorithm proposed in Armand and Omheni (Optim Methods Softw 32(1):1–21, 2017). A new parameter is introduced to scale the objective function and, in case of infeasibility, to force the convergence of the iterates to an infeasible stationary point. It is shown, under mild assumptions, that whenever the algorithm converges to an infeasible stationary point, the rate of convergence is quadratic. This is a new convergence result for the class of augmented Lagrangian methods. The global convergence of the algorithm is also analyzed. It is also proved that, when the algorithm converges to a stationary point, the properties of the original algorithm are preserved. The numerical experiments show that our new approach is as good as the original one when the algorithm converges to a local minimum, but much more efficient in case of infeasibility.

Nonlinear Stepsize Control Algorithms: Complexity Bounds for First- and Second-Order Optimality

A nonlinear stepsize control (NSC) framework has been proposed by Toint (Optim Methods Softw 28:82–95, 2013) for unconstrained optimization, generalizing several trust-region and regularization algorithms. More recently, worst-case complexity bounds to achieve approximate first-order optimality were proved by Grapiglia, Yuan and Yuan (Math Program 152:491–520, 2015) for the generic NSC framework. In this paper, improved complexity bounds for first-order optimality are obtained. Furthermore, complexity bounds for second-order optimality are also provided.

Maximum flows in generalized processing networks

Processing networks (cf. Koene in Minimal cost flow in processing networks: a primal approach, 1982) and manufacturing networks (cf. Fang and Qi in Optim Methods Softw 18:143---165, 2003) are well-studied extensions of traditional network flow problems that allow to model the decomposition or distillation of products in a manufacturing process. In these models, so called flow ratios $$\alpha _e \in [0,1]$$źeź[0,1] are assigned to all outgoing edges of special processing nodes. For each such special node, these flow ratios, which are required to sum up to one, determine the fraction of the total outgoing flow that flows through the respective edges. In this paper, we generalize processing networks to the case that these flow ratios only impose an upper bound on the respective fractions and, in particular, may sum up to more than one at each node. We show that a flow decomposition similar to the one for traditional network flows is possible and can be computed in strongly polynomial time. Moreover, we show that there exists a fully polynomial-time approximation scheme (FPTAS) for the maximum flow problem in these generalized processing networks if the underlying graph is acyclic and we provide two exact algorithms with strongly polynomial running-time for the problem on series---parallel graphs. Finally, we study the case of integral flows and show that the problem becomes $${\mathcal {NP}}$$NP-hard to solve and approximate in this case.

Capitalizing on live variables: new algorithms for efficient Hessian computation via automatic differentiation

We revisit an algorithm [called Edge Pushing (EP)] for computing Hessians using Automatic Differentiation (AD) recently proposed by Gower and Mello (Optim Methods Softw 27(2): 233–249, 2012). Here we give a new, simpler derivation for the EP algorithm based on the notion of live variables from data-flow analysis in compiler theory and redesign the algorithm with close attention to general applicability and performance. We call this algorithm Livarh and develop an extension of Livarh that incorporates preaccumulation to further reduce execution time—the resulting algorithm is called Livarhacc. We engineer robust implementations for both algorithms Livarh and Livarhacc within ADOL-C, a widely-used operator overloading based AD software tool. Rigorous complexity analyses for the algorithms are provided, and the performance of the algorithms is evaluated using a mesh optimization application and several kinds of synthetic functions as testbeds. The results show that the new algorithms outperform state-of-the-art sparse methods (based on sparsity pattern detection, coloring, compressed matrix evaluation, and recovery) in some cases by orders of magnitude. We have made our implementation available online as open-source software and it will be included in a future release of ADOL-C.

Finite element error estimates for an optimal control problem governed by the Burgers equation

We derive a-priori error estimates for the finite-element approximation of a distributed optimal control problem governed by the steady one-dimensional Burgers equation with pointwise box constraints on the control. Here the approximation of the state and the control is done by using piecewise linear functions. With this choice, a superlinear order of convergence for the control is obtained in the $$L^2$$L2-norm; moreover, under a further assumption on the regularity structure of the optimal control this error estimate can be improved to $$h^{3/2}$$h3/2, extending the results in Rosch (Optim. Methods Softw. 21(1): 121---134, 2006). The theoretical findings are tested experimentally by means of numerical examples.

Three modified Polak-Ribière-Polyak conjugate gradient methods with sufficient descent property

In this paper, three modified Polak-Ribiere-Polyak (PRP) conjugate gradient methods for unconstrained optimization are proposed. They are based on the two-term PRP method proposed by Cheng (Numer. Funct. Anal. Optim. 28:1217-1230, 2007), the three-term PRP method proposed by Zhang et al. (IMA J. Numer. Anal. 26:629-640, 2006), and the descent PRP method proposed by Yu et al. (Optim. Methods Softw. 23:275-293, 2008). These modified methods possess the sufficient descent property without any line searches. Moreover, if the exact line search is used, they reduce to the classical PRP method. Under standard assumptions, we show that these three methods converge globally with a Wolfe line search. We also report some numerical results to show the efficiency of the proposed methods.

Analysis of copositive optimization based linear programming bounds on standard quadratic optimization

The problem of minimizing a quadratic form over the unit simplex, referred to as a standard quadratic optimization problem, admits an exact reformulation as a linear optimization problem over the convex cone of completely positive matrices. This computationally intractable cone can be approximated in various ways from the inside and from the outside by two sequences of nested tractable convex cones of increasing accuracy. In this paper, we focus on the inner polyhedral approximations due to Yildirim (Optim Methods Softw 27(1):155---173, 2012) and the outer polyhedral approximations due to de Klerk and Pasechnik (SIAM J Optim 12(4):875---892, 2002). We investigate the sequences of upper and lower bounds on the optimal value of a standard quadratic optimization problem arising from these two hierarchies of inner and outer polyhedral approximations. We give complete algebraic descriptions of the sets of instances on which upper and lower bounds are exact at any given finite level of the hierarchy. We identify the structural properties of the sets of instances on which upper and lower bounds converge to the optimal value only in the limit. We present several geometric and topological properties of these sets. Our results shed light on the strengths and limitations of these inner and outer polyhedral approximations in the context of standard quadratic optimization.

On the convergence and worst-case complexity of trust-region and regularization methods for unconstrained optimization

A nonlinear stepsize control framework for unconstrained optimization was recently proposed by Toint (Optim Methods Softw 28:82---95, 2013), providing a unified setting in which the global convergence can be proved for trust-region algorithms and regularization schemes. The original analysis assumes that the Hessians of the models are uniformly bounded. In this paper, the global convergence of the nonlinear stepsize control algorithm is proved under the assumption that the norm of the Hessians can grow by a constant amount at each iteration. The worst-case complexity is also investigated. The results obtained for unconstrained smooth optimization are extended to some algorithms for composite nonsmooth optimization and unconstrained multiobjective optimization as well.