Nonsmooth Optimization Methods Research Articles

Nonsmooth µ‐synthesis

AbstractWe revisit robust complex‐ and mixed‐ µ‐synthesis problems based on upper bounds and show that they can be recast as specially structured controller design programs. The proposed reformulations suggest a streamlined handling of µ‐synthesis problems using recently developed (local) nonsmooth optimization methods, where both scalings or multipliers and a controller of given structure are obtained simultaneously. A first cut of the nonsmooth programming software for structured H∞ synthesis is made available through the MATLAB R2010b Prerelease, Robust Control Toolbox Version 3.5 developed by The MathWorks, Inc. Copyright © 2010 John Wiley & Sons, Ltd.

Nonsmooth optimization: theory and algorithms

This is a summary of the author’s PhD thesis supervised by Manlio Gaudioso and Maria Flavia Monaco and defended on 21 February 2008 at the Universita della Calabria. The thesis is a survey on nonsmooth optimization methods for both convex and nonconvex functions. The main contribution of the dissertation is the presentation of a new bundle type method. The thesis is written in English and is available from http://www2.deis.unical.it/logilab/gorgone.

Improved Convergence Result for the Discrete Gradient and Secant Methods for Nonsmooth Optimization

We study a generalization of the nonderivative discrete gradient method of Bagirov et al. for minimizing a locally Lipschitz function f on ℝn. We strengthen the existing convergence result for this method by showing that it either drives the f-values to −∞ or each of its cluster points is Clarke stationary for f, without requiring the compactness of the level sets of f. Our generalization is an approximate bundle method, which also subsumes the secant method of Bagirov et al.

Total Variation Projection With First Order Schemes

This article proposes a new algorithm to compute the projection on the set of images whose total variation is bounded by a constant. The projection is computed through a dual formulation that is solved by first order non-smooth optimization methods. This yields an iterative algorithm that applies iterative soft thresholding to the dual vector field, and for which we establish convergence rate on the primal iterates. This projection algorithm can then be used as a building block in a variety of applications such as solving inverse problems under a total variation constraint, or for texture synthesis. Numerical results are reported to illustrate the usefulness and potential applicability of our TV projection algorithm on various examples including denoising, texture synthesis, inpainting, deconvolution and tomography problems. We also show that our projection algorithm competes favorably with state-of-the-art TV projection methods in terms of convergence speed.

Models and information technologies for decision support during structural and technological changes

Models, numerical algorithms, and database and software components aimed at decision support during the elaboration of energy-saving measures are considered. Modern methods of nonsmooth optimization are applied to solve relevant optimization problems.

Auxiliary Principle and Iterative Algorithms for Lions-Stampacchia Variational Inequalities

In this paper, we extend the auxiliary principle (Cohen in J. Optim. Theory Appl. 49:325---333, 1988) to study a class of Lions-Stampacchia variational inequalities in Hilbert spaces. Our method consists in approximating, in the subproblems, the nonsmooth convex function by a sequence of piecewise linear and convex functions, as in the bundle method for nonsmooth optimization. This makes the subproblems more tractable. We show the existence of a solution for this Lions-Stampacchia variational inequality and explain how to build a new iterative scheme and a new stopping criterion. This iterative scheme and criterion are different from those commonly used in the special case of nonsmooth optimization. We study also the convergence of iterative sequences generated by the algorithm.

A merit function approach to the subgradient method with averaging

We consider a version of the subgradient method for convex nonsmooth optimization involving subgradient averaging. Using a merit function approach in the space of decisions and subgradient estimates, we prove convergence of the primal variables to an optimal solution and of the dual variables to an optimal subgradient. Application to dual convex optimization problems is discussed.

On solving the Lagrangian dual of integer programs via an incremental approach

The Lagrangian dual of an integer program can be formulated as a min-max problem where the objective function is convex, piecewise affine and, hence, nonsmooth. It is usually tackled by means of subgradient algorithms, or multiplier adjustment techniques, or even more sophisticated nonsmooth optimization methods such as bundle-type algorithms. Recently a new approach to solving unconstrained convex finite min-max problems has been proposed, which has the nice property of working almost independently of the exact evaluation of the objective function at every iterate-point. In the paper we adapt the method, which is of the descent type, to the solution of the Lagrangian dual. Since the Lagrangian relaxation need not be solved exactly, the approach appears suitable whenever the Lagrangian dual must be solved many times (e.g., to improve the bound at each node of a branch-and-bound tree), and effective heuristic algorithms at low computational cost are available for solving the Lagrangian relaxation. We present an application to the Generalized Assignment Problem (GAP) and discuss the results of our numerical experimentation on a set of standard test problems.

Implementation of novel methods of global and nonsmooth optimization: GANSO programming library§

We discuss the implementation of a number of modern methods of global and nonsmooth continuous optimization, based on the ideas of Rubinov, in a programming library GANSO. GANSO implements the derivative-free bundle method, the extended cutting angle method, dynamical system-based optimization and their various combinations and heuristics. We outline the main ideas behind each method, and report on the interfacing with Matlab and Maple packages. §We dedicate this contribution to the memory of our dear friend and teacher Alex Rubinov, who was an inspiration for so many people around the globe.

On some properties and an application of the logarithmic barrier method

We analyze the logarithmic barrier method for nonsmooth convex optimization in the setting of point-to-set theory. This general framework allows us to both extend and include classical results. We also propose an application for finding efficient points of nonsmooth constrained convex vector-valued problems.

IQC analysis and synthesis via nonsmooth optimization

Integral quadratic constraints (IQCs) are used in system theory to model nonlinear phenomena within the framework of linear feedback control. IQC theory addresses parametric robustness, saturation effects, sector nonlinearities, passivity, and much else. In IQC analysis specially structured linear matrix inequalities (LMIs) arise and are currently addressed by structure exploiting LMI solvers. Controller synthesis under IQC constraints is nonconvex and much harder and has been attempted sporadically by global optimization techniques such as branch and bound, cutting plane or D - K -type coordinate descent ideas. Here, we revisit IQC theory and propose a completely different algorithmic solution based on local and nonsmooth optimization methods. This is less ambitious than global methods, but is very promising in practice. Our approach, while aiming high at IQC synthesis, offers new answers even for IQC analysis, because we optimize without Lyapunov variables. For high-order systems this leads to a significant reduction of the number of unknowns.

Globally convergent limited memory bundle method for large-scale nonsmooth optimization

Many practical optimization problems involve nonsmooth (that is, not necessarily differentiable) functions of thousands of variables. In the paper [Haarala, Miettinen, Makela, Optimization Methods and Software, 19, (2004), pp. 673–692] we have described an efficient method for large-scale nonsmooth optimization. In this paper, we introduce a new variant of this method and prove its global convergence for locally Lipschitz continuous objective functions, which are not necessarily differentiable or convex. In addition, we give some encouraging results from numerical experiments.

Nonsmooth Optimization for Multidisk H∞ Synthesis

We discuss output feedback control design with multiple performance specifications, each measured in a weighted H∞-norm. The multidisk design problem consists in finding a stabilizing output feedback controller which minimizes the different performance specifications simultaneously using a worst case strategy. This is less conservative than existing approaches, but difficult to solve algorithmically due to inherent nonsmoothness and nonconvexity. We present a nonsmooth optimization method suited for the multidisk problem and more generally, for programs where the maximum of an infinite family of nonconvex maximum eigenvalue functions is minimized. The method is shown to perform well on a control problem for a helicopter at hover.

Trust region method for nonsmooth convex optimization

We propose an iterative method that solves a nonsmoothconvex optimization problem by converting the originalobjective function to a once continuously differentiablefunction by way of Moreau-Yosida regularization.The proposed method makes use of approximate functionand gradient values of the Moreau-Yosida regularizationinstead of the corresponding exact values.Under this setting, Fukushima and Qi (1996) and Raufand Fukushima (2000) proposed a proximal Newton method anda proximal BFGS method, respectively, for nonsmooth convex optimization.While these methods employ a line search strategyto achieve global convergence, the method proposed in this paperuses a trust region strategy.We establish global and superlinear convergence of the methodunder appropriate assumptions.

A Bundle Method for Solving Variational Inequalities

In this paper, we present a bundle method for solving a generalized variational inequality problem. This problem consists of finding a zero of the sum of two multivalued operators defined on a real Hilbert space. The first one, F, is monotone and the second is the subdifferential of a lower semicontinuous proper convex function. Our method is based on the auxiliary problem principle due to Cohen, and our strategy is to approximate, in the subproblems, the nonsmooth convex function by a sequence of convex piecewise linear functions, as in the bundle method for nonsmooth optimization. This makes the subproblems more tractable. First, we explain how to build, step by step, suitable piecewise linear approximations by means of a bundle strategy, and we present a new stopping criterion to determine whether the current approximation is good enough. This criterion is the same as that commonly used in the special case of nonsmooth optimization. Second, we study the convergence of the algorithm for the case when the stepsizes are chosen going to zero and for the case bounded away from zero. In the first case, the convergence can be proved under rather mild assumptions: the operator F is paramonotone and possibly multivalued. In the second case, the convergence needs a stronger assumption: F is single-valued and satisfies a Dunn property. Finally, we illustrate the behavior of the proposed algorithm by some numerical tests.

On Approximations with Finite Precision in Bundle Methods for Nonsmooth Optimization

We consider the proximal form of a bundle algorithm for minimizing a nonsmooth convex function, assuming that the function and subgradient values are evaluated approximately. We show how these approximations should be controlled in order to satisfy the desired optimality tolerance. For example, this is relevant in the context of Lagrangian relaxation, where obtaining exact information about the function and subgradient values involves solving exactly a certain optimization problem, which can be relatively costly (and as we show, in any case unnecessary). We show that approximation with some finite precision is sufficient in this setting and give an explicit characterization of this precision. Alternatively, our result can be viewed as a stability analysis of standard proximal bundle methods, as it answers the following question: for a given approximation error, what kind of approximate solution can be obtained and how does it depend on the magnitude of the perturbation?

Optimal control of flow with discontinuities

Optimal control of the 1-D Riemann problem of Euler equations is studied, with the initial values for pressure and density as control parameters. The least-squares type cost functional employs either distributed observations in time or observations calculated at the end of the assimilation window. Existence of solutions for the optimal control problem is proven. Smooth and nonsmooth optimization methods employ the numerical gradient (respectively, a subgradient) of the cost functional, obtained from the adjoint of the discrete forward model. The numerical flow obtained with the optimal initial conditions obtained from the nonsmooth minimization matches very well with the observations. The algorithm for smooth minimization converges for the shorter time horizon but fails to perform satisfactorily for the longer time horizon, except when the observations corresponding to shocks are detected and removed.

Applications of interval arithmetic in non-smooth global optimization

In this paper, an expansion scheme for general functions is presented, which can be used to achieve better enclosure of function ranges. In the context of interval branch-and-bound methods for non-smooth global optimization, a pruning technique is given via the expansion. An interval pruning test is established. This pruning test offers the possibility to cut away a large part of the currently investigated box by the optimization algorithm, which can be utilized as an accelerating device similar to the monotonicity test frequently used in interval methods for smooth problems. Numerical computation shows that the proposed method is reliable and can find all solutions of global optimization problem.

CONTINUOUS SUBDIFFERENTIAL APPROXIMATIONS AND THEIR APPLICATIONS

In this paper, we study continuous approximations to the Clarke subdifferential and the Demyanov– Rubinov quasidifferential. Different methods have been proposed and discussed for the construction of the continuous approximations. Numerical methods for minimization of the locally Lipschitzian functions which are based on the continuous approximations are described and their convergence is studied. To test the proposed methods, numerical experiments have been carried out and discussed in the paper. This paper presents some methods for the continuous approximation of the Clarke subdifferential and the Demyanov–Rubinov quasidifferential of a locally Lipschitzian function. On the basis of the continuous approximations, we propose numerical methods of nonsmooth optimization. The paper presents a survey of some results obtained in [2–10]. The notions of the Clarke subdifferential and the Demyanov–Rubinov quasidifferential play a key role in nonsmooth and nonconvex optimization. The Clarke subdifferential is subject to a calculus in the form of inclusions and the calculus cannot be used in general for the estimation of the subgradients. Unlike the Clarke subdifferential, the quasidifferential has a full-scale calculus, which can be used for its calculation, although its entire calculation requires some operations with polytopes in the n-dimensional space. The number of these polytopes and their vertices can be very large and, therefore, the calculation of the quasidifferential becomes very complicated. In smooth optimization, there exist minimization methods, which, instead of using the gradient, use its approximations through finite differences (forward, backward, and central differences). In [37], a very simple convex nondifferentiable function was presented, for which these finite differences may give no information about the subdifferential. It follows that these finite-difference estimates of the gradient cannot be used for the approximation of the subgradient of the nonsmooth functions. In the past decades, a few methods for the numerical computation of the subgradients were proposed and studied. In [55], Shor proposed special finite differences, which allow one to approximate one subgradient of convex functions. In [56, 57], Studniarski modified Shor’s algorithm and extended it to the class of subregular functions. In [11], Borwein proposed a method for the calculation of one subgradient of the convex functions. Finally, in [46], Nesterov introduced the notion of the lexicographically smooth functions and proposed a method for the calculation of the subgradients. In this paper, we introduce the notion of a discrete gradient as a certain version of finite-difference subgradient estimates. The discrete gradient is defined with respect to a given direction and thus allows one to approximate a directional derivative of a given function. This approximation allows us to propose an effective algorithm for the calculation of a descent direction of a function at a given point. In general, the set of discrete gradients approximates the entire subdifferential, which can present only theoretical interest. For the computation of the descent direction, we need to calculate only a few discrete gradients at a given point, and the proposed algorithm for such a computation is definitive. The lack of continuity of the subdifferential and quasidifferential mappings creates difficulties in the study of methods for the minimization of locally Lipschitz functions. In [59], it was noted that the lack of this property was responsible for the failure of nonsmooth steepest descent algorithms. On

Survey of Bundle Methods for Nonsmooth Optimization

Bundle methods are at the moment the most efficient and promising methods for nonsmooth optimization. They have been successfully used in many practical applications, for example, in economics, mechanics, engineering and optimal control. The aim of this paper is to give an overview of the development and history of the bundle methods from the seventies to the present. For simplicity, we first concentrate on the convex unconstrained case with a single objective function. The methods are later extended to nonconvex, constrained and multicriteria cases.