Nonsmooth Optimization Techniques Research Articles

On Shor's r-Algorithm for Problems with Constraints

Introduction. Nonsmooth optimization problems arise in a wide range of applications, including engineering, finance, and deep learning, where activation functions often have discontinuous derivatives, such as ReLU. Conventional optimization algorithms developed primarily for smooth problems face difficulties when applied to nonsmooth contexts due to discontinuities and other associated irregularities. Possible approaches to overcome these problems include smoothing of functions and applying non-smooth optimization techniques. In particular, Shor's r-algorithm (Shor, Zhurbenko (1971), Shor (1979)) with space stretching in the direction of the difference of two subsequent subgradients is a competitive non-smooth optimization method (Bagirov et al. (2014)). However, the original r-algorithm is designed to minimize unconstrained convex functions. The goal of the work. The standard technique for applying this algorithm to problems with constraints consists in the use of exact non-smooth penalty functions (Eremin (1967), Zangwill (1967)). At the same time, it is necessary to correctly choose (quite large) the penalty coefficient of the penalty functions. Norkin (2020, 2022), Galvan et al. (2021) propose the so-called projective exact penalty functions method, which theoretically does not require choice of the penalty coefficient. The purpose of the present work is to study an applicability of the new exact projective non-smooth penalty functions method for solving conditional problems of non-smooth optimization by Shor's r-algorithm. The results. In this paper, the original optimization problem with convex constraints is first transformed into an unconstrained problem by the projective penalty function method, and then the r-algorithm is used to solve the transformed problem. The results of testing this approach on problems with linear constraints using a program implemented in Matlab are presented. The results of the present study show that the standard method of non-smooth penalties combined with Shor's r-algorithm is fast, due to the use of the provided program to calculate the subgradients, but it requires the correct selection of the penalty parameter. The projective penalty method is slow because in this study it uses finite differences to calculate the gradients, but it is quite stable with respect to the choice of the penalty parameter. Further research will be aimed at investigating the differential properties of the projection mapping and reducing the time of computing subgradients for account of parallel calculations. Keywords: Subgradient descent, constrained optimization, r-algorithm, exact projective penalty.

A Shape-Newton Approach to the Problem of Covering with Identical Balls

The problem of covering a region of the plane with a fixed number of minimum-radius identical balls is studied in the present work. An explicit construction of bi-Lipschitz mappings is provided to model small perturbations of the union of balls. This allows us to obtain analytical expressions for first- and second-order derivatives of the cost functional using nonsmooth shape optimization techniques under appropriate regularity assumptions. For regions defined as the union of disjoint convex polygons, algorithms based on Voronoi diagrams that do not rely on approximations are given to compute the derivatives. Extensive numerical experiments illustrate the capabilities and limitations of the introduced approach.

μ-Synthesis FO-PID for Twin Rotor Aerodynamic System

μ-synthesis is a NP-hard optimization problem based on the generalized Robust Control framework which manages to find a controller which fulfills both robust stability and robust performance. In order to solve such problems, nonsmooth optimization techniques are employed to find nearly-optimal parameters values. However, the free parameters available for tuning must be involved only in classical arithmetic operations, which leads to a problem for the fractional-order operator or for its integer-order approximation, exponential operations being involved. The main goal of the current article consists of presenting a possibility to integrate a fixed-structure multiple-input-multiple-output (MIMO) fractional-order proportional-integral-derivative (FO-PID) controller in the μ-synthesis optimization problem. The solution consists in a possibility to find a set of tunable parameters isomorphic with the fractional-order such that the coefficients involved in the approximation of the fractional element, along with the formulation of a fixed-structure mixed-sensitivity loop shaping μ-synthesis control problem. The proposed design procedure is applied to a twin rotor aerodynamic system (TRAS) using both MATLAB numerical simulation and practical experiments on laboratory scale equipment. Moreover, a comparison with the unstructured μ-synthesis is performed, highlighting the advantages of the proposed solution: simpler form and guaranteed robust stability and performance.

Theories for design and analysis of robust [formula omitted] fault detectors

This paper deals with model-based fault detection and isolation problems, for linear time invariant systems subject to a large class of uncertainties and disturbances. A new approach based on non-smooth optimization techniques, is proposed to synthesize a state–space realization of the fault detection filter, within the H∞/H− setting. A set of new criteria for robust fault detection performance analysis is too proposed, using the generalized structured singular value theory. The proposed theories are illustrated on a satellite example, under a large class of nonlinear dependent uncertainties. Through a deep analysis of the example, it is shown how the H∞/H− design – μg analysis tools can serve as a general theory, to solve fault detection problems.

Robust Group Subspace Recovery: A New Approach for Multi-Modality Data Fusion

Robust Subspace Recovery (RoSuRe) algorithm was recently introduced as a principled and numerically efficient algorithm that unfolds underlying Unions of Subspaces (UoS) structure, present in the data. The union of Subspaces (UoS) is capable of identifying more complex trends in data sets than simple linear models. We build on and extend RoSuRe to prospect the structure of different data modalities individually. We propose a novel multi-modal data fusion approach based on group sparsity which we refer to as Robust Group Subspace Recovery (RoGSuRe). Relying on a bi-sparsity pursuit paradigm and non-smooth optimization techniques, the introduced framework learns a new joint representation of the time series from different data modalities, respecting an underlying UoS model. We subsequently integrate the obtained structures to form a unified subspace structure. The proposed approach exploits the structural dependencies between the different modalities data to cluster the associated target objects. The resulting fusion of the unlabeled sensors' data from experiments on audio and magnetic data has shown that our method is competitive with other state of the art subspace clustering methods. The resulting UoS structure is employed to classify newly observed data points, highlighting the abstraction capacity of the proposed method.

Loop-Separation Control for Very Flexible Aircraft

In this research, a loop-separation concept (LSC) is applied to a very flexible aircraft. Appealing features of this control architecture include decoupled control logic and the use of high-order models for controller design, thus not requiring model order reduction. The LSC consists of two control loops. The inner loop employs a linear quadratic regulator capable of stabilizing the aircraft while controlling the shape of the half-wings independently from each other by artificially stiffening the structure. Structural dynamics can potentially be used in the design of the inner-loop regulator. The outer-loop design is based on rigid-body outputs. The outer loops for tracking velocity, heading, sideslip angle, and altitude are designed using nonsmooth optimization techniques. Numerical results show that structural feedback in the inner loop yields better regulation performance and reduced structural deformations. The autopilot is capable of attaining the commanded reference with low control energy while meeting the maneuver requirements and regulating the elastic deformations of the wing.

Reduced-Order $H_\infty$ Filter Design for Delta Operator Systems Over Multiple Frequency Intervals

In this article, a reduced-order $H_\infty$ filter is designed for a delta operator system over multiple frequency intervals. By nonsmooth optimization techniques, subdifferential analysis is generalized in delta domain for minimizing the maximum value of the $H_\infty$ performance over the multiple frequency intervals. Based on the subdifferential analysis, an efficient descent algorithm is developed to obtain the reduced-order $H_\infty$ filter. Compared with some existing algorithms, the proposed descent algorithm not only reduces computational complexities but also improves the $H_\infty$ performance. Numerical examples are presented to demonstrate effectiveness of the proposed descent algorithm.

Solving k-center problems involving sets based on optimization techniques

The continuous k-center problem aims at finding k balls with the smallest radius to cover a finite number of given points in $$\mathbb {R}^n$$. In this paper, we propose and study the following generalized version of the k-center problem: Given a finite number of nonempty closed convex sets in $$\mathbb {R}^n$$, find k balls with the smallest radius such that their union intersects all of the sets. Because of its nonsmoothness and nonconvexity, this problem is very challenging. Based on nonsmooth optimization techniques, we first derive some qualitative properties of the problem and then propose new algorithms to solve the problem. Numerical experiments are also provided to show the effectiveness of the proposed algorithms.

Fixed order non-smooth robust H∞ wide area damping controller considering load uncertainties

This paper presents the design of a fixed order non-smooth robust H∞ wide area damping controller for the multi machine power system considering load uncertainties. Dynamic response of loads has a significant effect on system stability and directly determines the stability margin of the operating point. The effect of load uncertainties is considered by different percentage and size of induction motors as dynamic load which was considered by unstructured uncertainty. The proposed method is based on shaping of the open loop transfer function in the frequency domain. This proposed method is robust with respect to multi-model and load type (dynamic, static) and parametric time-delay uncertainties. Non-smooth optimization techniques are developed to H∞ synthesis problem under additional structural constraints on the controller. This approach avoids using Lyapunov variables and therefore leads to moderate size optimization programs even for very large systems.The performance of the controller has been tested on two-area four-machine test system model and New England 39 bus system under various operating scenarios and faults. The effectiveness of the developed robust centralized wide area controller, designed based on H∞ fixed order loop shaping synthesis, is compared with a conventional power system stabilizer.

Clustering in large data sets with the limited memory bundle method

The aim of this paper is to design an algorithm based on nonsmooth optimization techniques to solve the minimum sum-of-squares clustering problems in very large data sets. First, the clustering problem is formulated as a nonsmooth optimization problem. Then the limited memory bundle method [Haarala et al., 2007] is modified and combined with an incremental approach to design a new clustering algorithm. The algorithm is evaluated using real world data sets with both the large number of attributes and the large number of data points. It is also compared with some other optimization based clustering algorithms. The numerical results demonstrate the efficiency of the proposed algorithm for clustering in very large data sets.

Solving minimax control problems via nonsmooth optimization

We address minimax optimal control problems with linear dynamics. Under convexity assumptions, by using non-smooth optimization techniques, we derive a set of optimality conditions for the continuous-time case. We define an approximated discrete-time problem where analogous conditions hold. One of them allows us to design an easily implementable descent method. We analyze its convergence and we show some preliminary numerical results.

A Derivative-Free Riemannian Powell's Method, Minimizing Hartley-Entropy-Based ICA Contrast.

Even though the Hartley-entropy-based contrast function guarantees an unmixing local minimum, the reported nonsmooth optimization techniques that minimize this nondifferentiable function encounter computational bottlenecks. Toward this, Powell's derivative-free optimization method has been extended to a Riemannian manifold, namely, oblique manifold, for the recovery of quasi-correlated sources by minimizing this contrast function. The proposed scheme has been demonstrated to converge faster than the related algorithms in the literature, besides the impressive source separation results in simulations involving synthetic sources having finite-support distributions and correlated images.

Fixed Low-Order Controller Design and H∞Optimization for Large-Scale Dynamical Systems

Large-scale linear time-invariant dynamical systems with inputs and outputs present major challenges for controller design. Model-order reduction has become popular in recent years, but controllers designed for reduced-order models may result in unstable closed-loop plants when applied to the larger-scale system. We investigate the practicality of fixed low-order controller design applied directly to large-scale continuous-time sparse systems. We assume that it is practical to compute the eigenvalues with largest real part of such systems using Matlab’s eigs, which requires only matrix-vector products, but that it is not possible to compute the norm using Matlab’s getPeakGain or SLICüt’s slinorm, which use the Boyd-Balakrishnan- Bruinsma-Steinbuch algorithm, requiring both Hamiltonian eigenvalue decompositions and singular value decompositions. Instead, we employ a recently developed efficient algorithm called Hybrid-Expansion-Contraction (HEC), which while not guaranteed to correctly compute the norm, finds, under certain assumptions, at least a local maximizer of the associated transfer function. Our controller design code uses nonsmooth optimization techniques first to attempt to stabilize the closed-loop system and then to minimize its norm proxy as computed by HEC. It is implemented in a new experimental Matlab code hifüüS, based on the public-domain hifoo toolbox first presented in ROCOND 2006, and will be made available for public use after further investigation and development.

Nonsmooth Optimization Techniques on Riemannian Manifolds

We present the notion of weakly metrically regular functions on manifolds. Then, a sufficient condition for a real valued function defined on a complete Rie- mannian manifold to be weakly metrically regular is obtained, and two optimization problems on Riemannian manifolds are considered. Moreover, we present a general- ization of the Palais-Smale condition for lower semicontinuous functions defined on manifolds. Then, we use this notion to obtain necessary conditions of optimality for a general minimization problem on complete Riemannian manifolds.

Regularization for inverse models in remote sensing

Quantitative remote sensing is an appropriate way to estimate atmospheric parameters and structural parameters and spectral component signatures of Earth surface cover type. Since the real physical system that couples the atmosphere, water and the land surface is complicated, its description requires a comprehensive set of parameters, so any practical physical model can only be approximated by a limited mathematical model. The pivotal problem for quantitative remote sensing is inversion. Inverse problems are typically ill-posed; they are characterized by: (C1) the solution may not exist; (C2) the dimension of the solution space may be infinite; (C3) the solution is not continuous with variations of the observations. These issues exist for nearly all inverse problems in geosciences and quantitative remote sensing. For example, when the observation system is band-limited or sampling is poor, i.e. few observations are available or directions are poorly located, the inversion process would be underdetermined, which leads to a multiplicity of the solutions, the large condition number of the normalized system, and significant noise propagation. Hence (C2) and (C3) would be the difficulties for quantitative remote sensing inversion. This paper will address the theory and methods from the viewpoint that the quantitative remote sensing inverse problems can be represented by kernel-based operator equations and solved by coupling regularization and optimization methods. In particular, I propose sparse and non-smooth regularization and optimization techniques for solving inverse problems in remote sensing. Numerical experiments are also made to demonstrate the applicability of our algorithms.

Comments on: Algorithms for linear programming with linear complementarity constraints

This paper addresses the solution of linear programming problems that further incorporate complementarity restrictions involving designated pairs of complementary variables, as well as its generalization where the objective function is a nonlinear continuously differentiable function. Such problems subsume mixed linear complementary problems, absolute value programs, the linear bilevel programming problem, and the bilinear programming problem as well as more general nonconvex quadratic programs, among many others. The existing research on these problems focuses on finding strongly stationary points, or more general B-stationary points, both of which are necessary characterizations for local minima. However, computing B-stationary points is more combinatorial since it involves characterizing solutions that turn out to be Karush–Kuhn–Tucker (KKT) points for an entire family of nonlinear programs. In special nondegenerate cases (where at least one of each complementary pair is nonzero in any complementary solution) or under a special linear independence constraint qualification, B-stationary points also turn out to be strongly stationary. Among several algorithms for computing such strongly stationary or B-stationary points, the most competitive method to-date is a complementary active-set algorithm (CASET) and its advanced variants. In several studies, it has been verified that this procedure works well, even for degenerate problems, outperforming other nonlinear programming approaches such as penalty methods, non-smooth optimization techniques, interior point methods, and sequential quadratic programming algorithms. The CASET algorithm requires an initial complementary feasible solution to start with, which can always be computed by using an enumerative method.

Using Pattern Search Methods for Minimizing Clustering Problems

Abstract —Clustering is one of an interesting data mining topicsthat can be applied in many fields. Recently, the problem of clusteranalysis is formulated as a problem of nonsmooth, nonconvex opti-mization, and an algorithm for solving the cluster analysis problembased on nonsmooth optimization techniques is developed. Thisoptimization problem has a number of characteristics that make itchallenging: it has many local minimum, the optimization variablescan be either continuous or categorical, and there are no exactanalytical derivatives. In this study we show how to apply a particularclass of optimization methods known as pattern search methodsto address these challenges. These methods do not explicitly usederivatives, an important feature that has not been addressed inprevious studies. Results of numerical experiments are presentedwhich demonstrate the effectiveness of the proposed method. Keywords —Clustering functions, Non-smooth Optimization, Non-convex Optimization, Pattern Search Method.

Stochastic Optimization Algorithms for Support Vector Machines Classification

In this paper, we consider the problem of semi-supervised binary classification by Support Vector Machines (SVM). This problem is explored as an unconstrained and non-smooth optimization task when part of the available data is unlabelled. We apply non-smooth optimization techniques to classification where the objective function considered is non-convex and non-differentiable and so difficult to minimize. We explore and compare the properties of Simulated Annealing and of Simultaneous Perturbation Stochastic Approximation (SPSA) algorithms (SPSA with the Lipschitz Perturbation Operator, SPSA with the Uniform Perturbation Operator, Standard Finite Difference Approximation) for semi-supervised SVM classification. Numerical results are given, obtained by running the proposed methods on several standard test problems drawn from the binary classification literature. The performance of the classifiers were evaluated by analyzing Receiver Operating Characteristics (ROC).

Nonsmooth Optimization of Systems with Varying Structure

Abstract Abstract A novel method based on the generalized gradient and nonsmooth optimization techniques called bundle methods is introduced to optimize the performance of a class of dynamic systems whose governing equations change depending on the values of the parameters, controls and the current state of the system.

Stochastic approximation algorithms for support vector machines semi-supervised binary classification

In this paper,we consider the problem of semi-supervisedbinary classificationby Support Vector Machines (SVM). This problem is explored as an unconstrained and non-smooth optimization task when part of the available data is unlabelled. We apply non-smooth optimization techniques to classification problems where the objective function considered is non-convex and non-differentiable and so difficult to minimize. We explore and compare the properties of Stochastic Approximation algorithms (Simultaneous Perturbation Stochastic Approximation (SPSA) with the Lipschitz Perturbation Operator, SPSA with the Uniform Perturbation Operator, and Standard Finite Difference Approximation) for semi-supervised SVM classification. We present some numerical results obtained by running the proposed methods on several standard test problems drawn from the binary classification literature.