Nonsmooth Optimization Algorithm Research Articles

Nonsmooth projection-free optimization with functional constraints

AbstractThis paper presents a subgradient-based algorithm for constrained nonsmooth convex optimization that does not require projections onto the feasible set. While the well-established Frank–Wolfe algorithm and its variants already avoid projections, they are primarily designed for smooth objective functions. In contrast, our proposed algorithm can handle nonsmooth problems with general convex functional inequality constraints. It achieves an $$\epsilon $$ ϵ -suboptimal solution in $$\mathcal {O}(\epsilon ^{-2})$$ O ( ϵ - 2 ) iterations, with each iteration requiring only a single (potentially inexact) Linear Minimization Oracle call and a (possibly inexact) subgradient computation. This performance is consistent with existing lower bounds. Similar performance is observed when deterministic subgradients are replaced with stochastic subgradients. In the special case where there are no functional inequality constraints, our algorithm competes favorably with a recent nonsmooth projection-free method designed for constraint-free problems. Our approach utilizes a simple separation scheme in conjunction with a new Lagrange multiplier update rule.

Fast high quality highlight removal using a simplified matrix iteration method

Recently, a global non-smooth optimization method was developed for efficiently eliminating specular highlights from single color image. Although this method was reported to be effective on both natural and medical images, its algorithm’s speed is slow due to the non-smoothness of the objective function. This paper proposes a global smooth optimization method, where two gradient regularization terms are used to describe diffuse and specular component features for texture details. A simplified optimality condition is established and thus a new iteration algorithm for specular highlight removal is proposed. By employing the gradient regularization of diffuse reflection, image texture detail can be preserved during specular highlight removal. Due to the smoothness of the objective function, the proposed smooth optimization algorithm has lower computational complexity and less storage requirement than the global non-smooth optimization algorithm. Furthermore, the proposed iteration algorithm is guaranteed to converge to a global optimal solution under a fixed step length. Experimental results demonstrate that the proposed smooth optimization algorithm is much faster than the non-smooth optimization algorithm. Moreover, the proposed highlight removal algorithm is more effective on benchmark natural images and real laparoscopic images than conventional highlight removal algorithms.

A Hybrid and Inexact Algorithm for Nonconvex and Nonsmooth Optimization

A Hybrid and Inexact Algorithm for Nonconvex and Nonsmooth Optimization

Smoothing algorithms for nonsmooth optimization over the Stiefel manifold with applications to the graph Fourier basis problem

In this paper, we consider a class of nonsmooth and nonconvex optimization problems over the Stiefel manifold where the objective function is the summation of a nonconvex smooth function and a nonsmooth Lipschitz continuous convex function composed with a linear mapping. Besides, we are interested in its application to the graph Fourier basis problem. We propose three numerical algorithms for solving this problem, by combining smoothing methods and some existing algorithms for smooth optimization over the Stiefel manifold. In particular, we approximate the aforementioned nonsmooth convex function by its Moreau envelope in our smoothing methods, and prove that the Moreau envelope has many favorable properties. Thanks to this and the scheme for updating the smoothing parameter, we show that any accumulation point of the solution sequence generated by the proposed algorithms is a stationary point of the original optimization problem. Numerical experiments on building graph Fourier basis are conducted to demonstrate the efficiency of the proposed algorithms.

The design of continuous-time accelerated algorithm for nonsmooth distributed optimization without sharing local decision variables

The design of continuous-time accelerated algorithm for nonsmooth distributed optimization without sharing local decision variables

A Superlinearly Convergent Subgradient Method for Sharp Semismooth Problems

Subgradient methods comprise a fundamental class of nonsmooth optimization algorithms. Classical results show that certain subgradient methods converge sublinearly for general Lipschitz convex functions and converge linearly for convex functions that grow sharply away from solutions. Recent work has moreover extended these results to certain nonconvex problems. In this work, we seek to improve the complexity of these algorithms by asking the following question. Is it possible to design a superlinearly convergent subgradient method? We provide a positive answer to this question for a broad class of sharp semismooth functions. Funding: The research of D. Davis was supported by the Division of Mathematical Sciences [Grant 2047637] and the Alfred P. Sloan Foundation.

A clustering heuristic to improve a derivative-free algorithm for nonsmooth optimization

In this paper we propose an heuristic to improve the performances of the recently proposed derivative-free method for nonsmooth optimization CS-DFN. The heuristic is based on a clustering-type technique to compute an estimate of Clarke’s generalized gradient of the objective function, obtained via calculation of the (approximate) directional derivative along a certain set of directions. A search direction is then calculated by applying a nonsmooth Newton-type approach. As such, this direction (as it is shown by the numerical experiments) is a good descent direction for the objective function. We report some numerical results and comparison with the original CS-DFN method to show the utility of the proposed improvement on a set of well-known test problems.

A Convex Formulation for Robust Control of Bilinear Positive Systems With Evolutionary Dynamics

In this article, optimal control of bilinear positive systems with uncertain evolutionary dynamics is studied. The first objective is to determine optimal control signals such that the effect of the disturbance signals on the outputs of the nominal dynamics is minimized in terms of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{H}_2$</tex-math> </inline-formula> norm performance index. Properties of the positive systems, Metzler matrices, and the exploitable structure of the problem are utilized to show that the underlying problem is representable as a standard convex optimization problem. Next, the optimal strategy is examined in the presence of polytopic uncertainties. In this case, the optimization problem is a semi-infinite min-max problem. It is shown that under some assumptions, the underlying problem is equivalent to a finite min-max problem, i.e., it suffices to check only the vertices of the polytopic set for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{H}_2$</tex-math> </inline-formula> norm minimization. The proposed approach for polytopic bilinear positive systems is non-iterative, does not involve parameter tuning or nonsmooth optimization algorithms, and can be solved using off-the-shelf solvers. The effectiveness of the proposed approach is investigated for the optimal combination therapy of human immunodeficiency virus (HIV).

Structured IQC Synthesis of Robust H2 Controllers in the Frequency Domain

The problem of robust controller synthesis for plants affected by structured uncertainty, captured by integral quadratic constraints, is discussed. The solution is optimized towards a worst-case white noise rejection specification, which is a generalization of the standard H2-norm to the robust setting including possibly non-LTI uncertainty. Arbitrary structural constraints can be imposed on the control solution, making this method suitable for distributed systems. The nonsmooth optimization algorithm used to solve the robust synthesis problem operates directly in the frequency domain, eliminating scalability issues for complex systems and providing local optimality certificates. The method is evaluated using a literature example and a real-world system using a novel implementation of a robust H2-performance bound.

Mathematical Models of M.V. Mykhalevych for Forecasting Structural and Technological Changes

Introduction. The identification of structural and technological disproportions that affect crisis phenomena in the economy and the analysis of ways to eliminate them require a wide application of quantitative research methods, in particular, mathematical modeling. “Input-Output” tables of Leontief turned out to be quite a convenient tool for analyzing these economic issues. In Leontief-type models, the matrix of technical coefficients (matrix of direct costs) is assumed to be known and calculated on the basis of statistical information from the “input-output” tables. M.V. Mykhalevych formulated the “inverse” problem: how to determine those structural and technological changes that would reduce the cost of production and thereby increase the incomes of end consumers and make the economy more dynamic. Or, in other words, how to choose or adjust technical coefficients to improve the properties of the economic process. This work is devoted to two optimization problems built on the basis of models of this type. The purpose of the article is to optimize the interdisciplinary planning of structural and technological changes. Results. Inverse models of the Leontief type for optimization of structural and technological transformations in economic systems are considered. These models are formulated in terms of nonlinear programming problems and include two objective functions for maximization: total consumer incomes and the “income growth–production growth” multiplier. Algorithms and software for solving these problems are presented. Numerical optimization procedures are based on Shor's r-algorithm. Conclusions. The use of inverse models of the Leontief type will allow choosing promising directions of structural and technological transformations in both the macro- and microeconomy. The proposed mathematical apparatus based on non-smooth optimization algorithms proved to be a sufficiently effective tool for solving appropriate optimization problems in practice. Keywords: structural and technological changes, inter-industry balance, Leontief model, “input-output” matrix, inverse Leontief-type models, non-smooth optimization algorithms, software.

A Model Independent Control Scheme for Electromagnetic Scanning Micromirrors

Due to the existence of hysteresis and unmeasured electromagnetic torque, it is not easy to establish an accurate model of an electromagnetic scanning micromirror (ESMM) system. Therefore, the use of model-based angle control strategies limits the large-scale application of ESMM chips in industrial manufacturing. In this article, a model independent control (MIC) method is proposed to control the deflection angle of ESMM, which can avoid the modeling process and facilitate the large-scale industrial application of ESMM chips. In the proposed approach, the controller design is not based on mathematical model of the micromirror but just using the measured input and output of the ESMM. To predict the behavior of ESMM with hysteresis, a model-free predictor based on the measured system I/O data is proposed. After that, an online model-free nonsmooth optimization algorithm is developed to search for the optimal control solution. Then, the stability of the MIC system is analyzed. At the end, the performance of the proposed method is verified through simulation and experimental results.

Escaping Strict Saddle Points of the Moreau Envelope in Nonsmooth Optimization

Recent work has shown that stochastically perturbed gradient methods can efficiently escape strict saddle points of smooth functions. We extend this body of work to nonsmooth optimization, by analyzing an inexact analogue of a stochastically perturbed gradient method applied to the Moreau envelope. The main conclusion is that a variety of algorithms for nonsmooth optimization can escape strict saddle points of the Moreau envelope at a controlled rate. The main technical insight is that many algorithms applied to the proximal subproblem yield directions that approximate the gradient of the Moreau envelope.

On the convergence analysis of a penalty algorithm for nonsmooth optimization and its performance for solving hard-sphere problems

On the convergence analysis of a penalty algorithm for nonsmooth optimization and its performance for solving hard-sphere problems

A Derivative-Free Line-Search Algorithm for Simulation-Driven Design Optimization Using Multi-Fidelity Computations

The paper presents a multi-fidelity extension of a local line-search-based derivative-free algorithm for nonsmooth constrained optimization (MF-CS-DFN). The method is intended for use in the simulation-driven design optimization (SDDO) context, where multi-fidelity computations are used to evaluate the objective function. The proposed algorithm starts using low-fidelity evaluations and automatically switches to higher-fidelity evaluations based on the line-search step length. The multi-fidelity algorithm is driven by a suitably defined threshold and initialization values for the step length, which are associated to each fidelity level. These are selected to increase the accuracy of the objective evaluations while progressing to the optimal solution. The method is demonstrated for a multi-fidelity SDDO benchmark, namely pertaining to the hull-form optimization of a destroyer-type vessel, aiming at resistance minimization in calm water at fixed speed. Numerical simulations are based on a linear potential flow solver, where seven fidelity levels are used selecting systematically refined computational grids for the hull and the free surface. The method performance is assessed varying the steplength threshold and initialization approach. Specifically, four MF-CS-DFN setups are tested, and the optimization results are compared to its single-fidelity (high-fidelity-based) counterpart (CS-DFN). The MF-CS-DFN results are promising, achieving a resistance reduction of about 12% and showing a faster convergence than CS-DFN. Specifically, the MF extension is between one and two orders of magnitude faster than the original single-fidelity algorithm. For low computational budgets, MF-CS-DFN optimized designs exhibit a resistance that is about 6% lower than that achieved by CS-DFN.

Linking Distributed Optimization Models for Food, Water, and Energy Security Nexus Management

Traditional integrated modeling (IM) is based on developing and aggregating all relevant (sub)models and data into a single integrated linear programming (LP) model. Unfortunately, this approach is not applicable for IM under asymmetric information (ASI), i.e., when “private” information regarding sectoral/regional models is not available, or it cannot be shared by modeling teams (sectoral agencies). The lack of common information about LP submodels makes LP methods inapplicable for integrated LP modeling. The aim of this paper is to develop a new approach to link and optimize distributed sectoral/regional optimization models, providing a means of decentralized cross-sectoral coordination in the situation of ASI. Thus, the linkage methodology enables the investigation of policies in interdependent systems in a “decentralized” fashion. For linkage, the sectoral/regional models do not need recoding or reprogramming. They also do not require additional data harmonization tasks. Instead, they solve their LP submodels independently and in parallel by a specific iterative subgradient algorithm for nonsmooth optimization. The submodels continue to be the same separate LP models. A social planner (regulatory agency) only needs to adjust the joint resource constraints to simple subgradient changes calculated by the algorithm. The approach enables more stable and resilient systems’ performance and resource allocation as compared to the independent policies designed by separate models without accounting for interdependencies. The paper illustrates the application of the methodology to link detailed energy and agricultural production planning models under joint constraints on water and land use.

On the Continuity of the Tangent Cone to the Determinantal Variety

Tangent and normal cones play an important role in constrained optimization to describe admissible search directions and, in particular, to formulate optimality conditions. They notably appear in various recent algorithms for both smooth and nonsmooth low-rank optimization where the feasible set is the set $\mathbb {R}_{\le r}^{m \times n}$ of all m × n real matrices of rank at most r. In this paper, motivated by the convergence analysis of such algorithms, we study, by computing inner and outer limits, the continuity of the correspondence that maps each $X \in \mathbb {R}_{\le r}^{m \times n}$ to the tangent cone to $\mathbb {R}_{\le r}^{m \times n}$ at X. We also deduce results about the continuity of the corresponding normal cone correspondence. Finally, we show that our results include as a particular case the a-regularity of the Whitney stratification of $\mathbb {R}_{\le r}^{m \times n}$ following from the fact that this set is a real algebraic variety, called the real determinantal variety.

An optimization-based algorithm for simultaneous shaping of poles and zeros for non-collocated vibration suppression

This article presents a control design method for simultaneous shaping of the poles and zeros of linear time-invariant systems, motivated by the application of non-collocated vibration suppression to flexible multi-body systems. An entire suppression of vibrations at a target mass for a given excitation frequencies can be recast into the problem of assigning zeros of the transfer function from the excitation force to the target mass' position. The design requirement of achieving sufficient damping in the closed loop system combined with suppressing vibrations at the target, leads to the minimization of the spectral abscissa function of the closed loop system as a function of the controller parameters, subject to zero location constraints. These constraints exhibit polynomial dependence on the controller parameters. We present two approaches to solve the optimization problem which are both based on constraint elimination followed by application of an algorithm for non-smooth unconstrained optimization. The design approach is applicable to delay-free models as well as time-delay models of retarded and neutral type. Simulations results illustrate its applicability to a spring-mass-damper system.

Convergence Rate Comparison of Proximal Algorithms for Non-Smooth Convex Optimization With an Application to Texture Segmentation

In this paper we provide a theoretical and numerical comparison of convergence rates of forward-backward, Douglas-Rachford, and Peaceman-Rachford algorithms for minimizing the sum of a convex proper lower semicontinuous function and a strongly convex differentiable function with Lipschitz continuous gradient. Our results extend the comparison made in [1], when both functions are smooth, to the context where only one is assumed differentiable. Optimal step-sizes and rates of the three algorithms are compared theoretically and numerically in the context of texture segmentation problem, obtaining very sharp estimations and illustrating the high efficiency of Peaceman-Rachford splitting.

A redistributed cutting plane bundle-type algorithm for multiobjective nonsmooth optimization

&lt;abstract&gt;&lt;p&gt;I construct a new cutting-plane model for approximating nonsmooth nonconvex functions in multiobjective optimization and propose a new bundle-type method with the help of an improvement function. The presented bundle method possesses three features. Firstly, the objective and constraint functions are approximated by a new cutting-plane model, which is a local convexification of the corresponding functions, instead of the entire approximation for the functions, as most bundle methods do. Secondly, the subgradients and values of the objective and constraint functions are computed approximately. In other words, approximate calculation is applied to the method, and the proposed algorithm is doubly approximate to some extent. Thirdly, the introduction of the improvement function eliminates the necessity of employing any scalarization, which is the usual method when dealing with multiobjective optimization. Under reasonable conditions satisfactory convergence results are obtained.&lt;/p&gt;&lt;/abstract&gt;

Robust integrated control/structure co-design for stratospheric balloons

Stratospheric balloons offer cost-effective platforms for optical payloads in the context of astronomy missions. During the 2018 flight of the Faint Intergalactic medium Redshifted Emission Balloon (FIREBall) experiment, the moon light was scattered from the surface of the balloon and re-directed into the telescope which resulted in degraded optical performance. To reduce this parasite effect, it is sought to increase the length of the fight train. However, this change in the mechanical design significantly modifies the dynamics of the system and the pointing performance must not be altered. In this purpose, a robust integrated control/structure co-design method is proposed. After deriving a Linear Fractional Transformation (LFT) model of the system, the co-design is tackled as a multi-objective, structured, robust H2/H∞ problem that is solved with a non-smooth optimization algorithm to maximize the train's length under constraints of pointing performance. By optimizing in a single iteration the controllers along with the structural parameter with regard to the worst-case configurations of the uncertain parameters, time-consuming procedures requiring not only to iterate between control and mechanical design, but also to analyze the robustness based on Monte-Carlo simulations, are avoided.