Nonsmooth Optimization Research Articles

Autonomous Tracking and State Estimation With Generalized Group Lasso

We address the problem of autonomous tracking and state estimation for marine vessels, autonomous vehicles, and other dynamic signals under a (structured) sparsity assumption. The aim is to improve the tracking and estimation accuracy with respect to the classical Bayesian filters and smoothers. We formulate the estimation problem as a dynamic generalized group Lasso problem and develop a class of smoothing-and-splitting methods to solve it. The Levenberg-Marquardt iterated extended Kalman smoother-based multiblock alternating direction method of multipliers (LM-IEKS-mADMMs) algorithms are based on the alternating direction method of multipliers (ADMMs) framework. This leads to minimization subproblems with an inherent structure to which three new augmented recursive smoothers are applied. Our methods can deal with large-scale problems without preprocessing for dimensionality reduction. Moreover, the methods allow one to solve nonsmooth nonconvex optimization problems. We then prove that under mild conditions, the proposed methods converge to a stationary point of the optimization problem. By simulated and real-data experiments, including multisensor range measurement problems, marine vessel tracking, autonomous vehicle tracking, and audio signal restoration, we show the practical effectiveness of the proposed methods.

A class of infeasible proximal bundle methods for nonsmooth nonconvex multi-objective optimization problems

A class of infeasible proximal bundle methods for nonsmooth nonconvex multi-objective optimization problems

Completely positive factorization by a Riemannian smoothing method

Copositive optimization is a special case of convex conic programming, and it consists of optimizing a linear function over the cone of all completely positive matrices under linear constraints. Copositive optimization provides powerful relaxations of NP-hard quadratic problems or combinatorial problems, but there are still many open problems regarding copositive or completely positive matrices. In this paper, we focus on one such problem; finding a completely positive (CP) factorization for a given completely positive matrix. We treat it as a nonsmooth Riemannian optimization problem, i.e., a minimization problem of a nonsmooth function over a Riemannian manifold. To solve this problem, we present a general smoothing framework for solving nonsmooth Riemannian optimization problems and show convergence to a stationary point of the original problem. An advantage is that we can implement it quickly with minimal effort by directly using the existing standard smooth Riemannian solvers, such as Manopt. Numerical experiments show the efficiency of our method especially for large-scale CP factorizations.

A semismooth Newton based augmented Lagrangian method for nonsmooth optimization on matrix manifolds

This paper is devoted to studying an augmented Lagrangian method for solving a class of manifold optimization problems, which have nonsmooth objective functions and nonlinear constraints. Under the constant positive linear dependence condition on manifolds, we show that the proposed method converges to a stationary point of the nonsmooth manifold optimization problem. Moreover, we propose a globalized semismooth Newton method to solve the augmented Lagrangian subproblem on manifolds efficiently. The local superlinear convergence of the manifold semismooth Newton method is also established under some suitable conditions. We also prove that the semismoothness on submanifolds can be inherited from that in the ambient manifold. Finally, numerical experiments on compressed modes and (constrained) sparse principal component analysis illustrate the advantages of the proposed method.

Perturbed Iterate SGD for Lipschitz Continuous Loss Functions

This paper presents an extension of stochastic gradient descent for the minimization of Lipschitz continuous loss functions. Our motivation is for use in non-smooth non-convex stochastic optimization problems, which are frequently encountered in applications such as machine learning. Using the Clarke \(\epsilon \)-subdifferential, we prove the non-asymptotic convergence to an approximate stationary point in expectation for the proposed method. From this result, a method with non-asymptotic convergence with high probability, as well as a method with asymptotic convergence to a Clarke stationary point almost surely are developed. Our results hold under the assumption that the stochastic loss function is a Carathéodory function which is almost everywhere Lipschitz continuous in the decision variables. To the best of our knowledge, this is the first non-asymptotic convergence analysis under these minimal assumptions.

DoG-HiT: A novel VLBI multiscale imaging approach

Context. Reconstructing images from very long baseline interferometry (VLBI) data with a sparse sampling of the Fourier domain (uv-coverage) constitutes an ill-posed deconvolution problem. It requires application of robust algorithms, maximizing the information extraction from all of the sampled spatial scales, and minimizing the influence of the unsampled scales on image quality. Aims. We develop a new multiscale wavelet deconvolution algorithm, DoG-HiT, for imaging sparsely sampled interferometric data, which combines the difference of Gaussian (DoG) wavelets and hard image thresholding (HiT). Based on DoG-HiT, we propose a multistep imaging pipeline for analysis of interferometric data. Methods. DoG-HiT applies the compressed sensing approach to imaging by employing a flexible DoG wavelet dictionary, which is designed to adapt smoothly to the uv-coverage. It uses closure properties as data fidelity terms only, initially, and performs nonconvex, nonsmooth optimization by an amplitude-conserving and total-flux-conserving, hard thresholding splitting. DoG-HiT calculates a multiresolution support as a side product. The final reconstruction is refined through self-calibration loops and imaging with amplitude and phase information applied for the multiresolution support only. Results. We demonstrate the stability of DoG-HiT, and benchmark its performance against image reconstructions made with the CLEAN and regularized maximum-likelihood (RML) methods using synthetic data. The comparison shows that DoG-HiT matches the super-resolution achieved by the RML reconstructions and surpasses the sensitivity to extended emission reached by CLEAN. Conclusions. The application of regularized maximum likelihood methods, outfitted with flexible multiscale wavelet dictionaries, to imaging of interferometric data, matches the performance of state-of-the art convex optimization imaging algorithms and requires fewer prior and user-defined constraints.

Decentralized Triple Proximal Splitting Algorithm With Uncoordinated Stepsizes for Nonsmooth Composite Optimization Problems

In this article, we consider a class of decentralized nonsmooth composite optimization problems over undirected graphs. The global optimization problem is to minimize the sum of local objective functions consisting of a Lipschitz-differentiable convex function and two possibly nonsmooth convex functions, one of which contains a bounded linear operator. The goal is to solve the global optimization problem through decentralized computation and communication over a network of agents without a central coordinator. Through using triple proximal splitting operators to deal with the nonsmooth terms, we come up with a novel decentralized algorithm with uncoordinated stepsizes, where the stepsizes with independent upper bounds are also distributed for agents or edges over the communication network. Furthermore, we establish the sublinear convergence rate for the proposed algorithm in terms of the first-order optimality residual in a nonergodic sense. Simulation experiments on a constrained quadratic programming problem and an optimal load-sharing problem are carried out to verify the correctness of the theoretical results.

Gradient Sampling Methods with Inexact Subproblem Solutions and Gradient Aggregation

Gradient sampling (GS) methods for the minimization of objective functions that may be nonconvex and/or nonsmooth are proposed, analyzed, and tested. One of the most computationally expensive components of contemporary GS methods is the need to solve a convex quadratic subproblem in each iteration. By contrast, the methods proposed in this paper allow the use of inexact solutions of these subproblems, which, as proved in the paper, can be incorporated without the loss of theoretical convergence guarantees. Numerical experiments show that, by exploiting inexact subproblem solutions, one can consistently reduce the computational effort required by a GS method. Additionally, a strategy is proposed for aggregating gradient information after a subproblem is solved (potentially inexactly) as has been exploited in bundle methods for nonsmooth optimization. It is proved that the aggregation scheme can be introduced without the loss of theoretical convergence guarantees. Numerical experiments show that incorporating this gradient aggregation approach can also reduce the computational effort required by a GS method.

Power Allocation in Smart Grid Aided Cell Free Massive MIMO Systems for URLLC Applications

This paper investigates power allocation in cell-free massive MIMO for URLLC applications, where each access point is powered by smart grid or solar panels/wind-turbines. In particular, the closed-form achievable rate for URRLC is derived. By exploiting the derived results, a non-convex and non-smooth optimization problem is formulated for maximizing the minimum URRLC rate of all multicast group, and is NP-hard. To deal with this problem, the formulated optimization problem is transformed into a tractable version, then the classical CVX tool is adopted to solve the resultant problem. Numerical simulation verifies the practicability of the derived theoretical results and the proposed algorithm.

A Model Independent Control Scheme for Electromagnetic Scanning Micromirrors

Due to the existence of hysteresis and unmeasured electromagnetic torque, it is not easy to establish an accurate model of an electromagnetic scanning micromirror (ESMM) system. Therefore, the use of model-based angle control strategies limits the large-scale application of ESMM chips in industrial manufacturing. In this article, a model independent control (MIC) method is proposed to control the deflection angle of ESMM, which can avoid the modeling process and facilitate the large-scale industrial application of ESMM chips. In the proposed approach, the controller design is not based on mathematical model of the micromirror but just using the measured input and output of the ESMM. To predict the behavior of ESMM with hysteresis, a model-free predictor based on the measured system I/O data is proposed. After that, an online model-free nonsmooth optimization algorithm is developed to search for the optimal control solution. Then, the stability of the MIC system is analyzed. At the end, the performance of the proposed method is verified through simulation and experimental results.

Spectral projected subgradient method for nonsmooth convex optimization problems

We consider constrained optimization problems with a nonsmooth objective function in the form of mathematical expectation. The Sample Average Approximation (SAA) is used to estimate the objective function and variable sample size strategy is employed. The proposed algorithm combines an SAA subgradient with the spectral coefficient in order to provide a suitable direction which improves the performance of the first order method as shown by numerical results. The step sizes are chosen from the predefined interval and the almost sure convergence of the method is proved under the standard assumptions in stochastic environment. To enhance the performance of the proposed algorithm, we further specify the choice of the step size by introducing an Armijo-like procedure adapted to this framework. Considering the computational cost on machine learning problems, we conclude that the line search improves the performance significantly. Numerical experiments conducted on finite sum problems also reveal that the variable sample strategy outperforms the full sample approach.

Fast and scalable learning of sparse changes in high-dimensional graphical model structure

We focus on the problem of estimating the change in the dependency structures of two p-dimensional Gaussian Graphical models (GGMs). Previous studies for sparse change estimation in GGMs involve expensive and difficult non-smooth optimization. We propose a novel method, DIFFEE for estimating DIFFerential networks via an Elementary Estimator under a high-dimensional situation. DIFFEE is solved through a faster and closed-form solution that enables it to work in large-scale settings. Notice that GGM assumes data are generated from a Gaussian distribution. However, the Gaussian assumption is too strict and can not be satisfied with all the real-world data generated from a complex process. Therefore, we further extend DIFFEE to NPN-DIFFEE by assuming that data are drawn from the nonparanormal distribution (a large family of distributions) instead of a multivariate Gaussian distribution. Thus, NPN-DIFFEE is applicable to more general conditions. We conduct a rigorous statistical analysis showing that surprisingly DIFFEE achieves the same asymptotic convergence rates as the state-of-the-art estimators that are much more difficult to compute. Our experimental results on multiple synthetic datasets and one real-world data about brain connectivity show strong performance improvements over baselines, as well as significant computational benefits.

Non-smooth variational problems and applications.

Mathematical methods based on the variational approach are successfully used in a broad range of applications, especially those fields that are oriented on partial differential equations. Our problem area addresses a wide class of nonlinear variational problems described by all kinds of static and evolution equations, inverse and ill-posed problems, non-smooth and non-convex optimization, and optimal control including shape and topology optimization. Within these directions, we focus but are not limited to singular and unilaterally constrained problems arising in mechanics and physics, which are governed by complex systems of generalized variational equations and inequalities. Whereas classical mathematical tools are not applicable here, we aim at a non-standard well-posedness analysis, numerical methods, asymptotic and approximation techniques including homogenization, which are successful within the primal as well as the dual variational formalism. In a broad scope, the theme issue objectives are directed toward advances that are attained in the mathematical theory of non-smooth variational problems, its physical consistency, numerical simulation and application to engineering sciences.This article is part of the theme issue ‘Non-smooth variational problems and applications’.

A new randomized primal-dual algorithm for convex optimization with fast last iterate convergence rates

We develop a novel unified randomized block-coordinate primal-dual algorithm to solve a class of nonsmooth constrained convex optimization problems, which covers different existing variants and model settings from the literature. We prove that our algorithm achieves and convergence rates in two cases: merely convexity and strong convexity, respectively, where k is the iteration counter and n is the number of block-coordinates. These rates are known to be optimal (up to a constant factor) when n = 1. Our convergence rates are obtained through three criteria: primal objective residual and primal feasibility violation, dual objective residual, and primal-dual expected gap. Moreover, our rates for the primal problem are on the last-iterate sequence. Our dual convergence guarantee requires additionally a Lipschitz continuity assumption. We specify our algorithm to handle two important special cases, where our rates are still applied. Finally, we verify our algorithm on two well-studied numerical examples and compare it with two existing methods. Our results show that the proposed method has encouraging performance on different experiments.

Optimally Safe Tank Changeover Operation Using a Smooth Optimization Formulation.

Tank changeover is a routine process in industry for placing fuel tanks into or out of service. The operation must use inert gas to avoid the flammability zone. However, inert gas consumption should be minimized for economic reasons. This requires dynamic modeling and optimization of the process, as addressed in the present work. A new dynamic optimization problem for minimizing the inert gas consumption, while ensuring fire safety is proposed. As part of the problem constraints, the flammability zone is characterized by disjunctive constraints, which are then converted to a new simple, nonsmooth formula, removing the need for data regression. This together with the multi-mode flow equations in the model leads to a nonsmooth dynamic optimization problem. To enable reliable solution by gradient-based solvers, the problem is reformulated to a smooth one using sigmoid functions. Case studies of methane tank purging and filling operations demonstrate that the proposed approach is able to minimize the inert consumption by providing optimal trajectories of the tank inlet and outlet flow rates, while ensuring the operation remains outside the flammability zone. It is shown that the proposed dynamic optimization can yield significant economic benefits as it reduces the nitrogen consumption by about two-third in one of the examples solved.

Optimality and Duality of Approximate Quasi Weakly Efficient Solution for Nonsmooth Vector Optimization Problems

This paper aims at studying optimality conditions and duality theorems of an approximate quasi weakly efficient solution for a class of nonsmooth vector optimization problems (VOP). First, a necessary optimality condition to the problem (VOP) is established by using the Clarke subdifferential. Second, the concept of approximate pseudo quasi type-I function is introduced, and under its hypothesis, a sufficient optimality condition to the problem (VOP) is also obtained. Finally, the approximate Mond–Weir dual model of the problem (VOP) is presented, and then, weak, strong, and converse duality theorems are established.

Proximal ADMM for nonconvex and nonsmooth optimization

By enabling the nodes or agents to solve small-sized subproblems to achieve coordination, distributed algorithms are favored by many networked systems for efficient and scalable computation. While for convex problems, substantial distributed algorithms are available, the results for the more broad nonconvex counterparts are extremely lacking. This paper develops a distributed algorithm for a class of nonconvex and nonsmooth problems featured by (i) a nonconvex objective formed by both separate and composite components regarding the decision variables of interconnected agents, (ii) local bounded convex constraints, and (iii) coupled linear constraints. This problem is directly originated from smart buildings and is also broad in other domains. To provide a distributed algorithm with convergence guarantee, we revise the powerful alternating direction method of multiplier (ADMM) method and proposed a proximal ADMM. Specifically, noting that the main difficulty to establish the convergence for the nonconvex and nonsmooth optimization with ADMM is to assume the boundness of dual updates, we propose to update the dual variables in a discounted manner. This leads to the establishment of a so-called sufficiently decreasing and lower bounded Lyapunov function, which is critical to establish the convergence. We prove that the method converges to some approximate stationary points. We besides showcase the efficacy and performance of the method by a numerical example and the concrete application to multi-zone heating, ventilation, and air-conditioning (HVAC) control in smart buildings.

Decentralized Primal-Dual Proximal Operator Algorithm for Constrained Nonsmooth Composite Optimization Problems over Networks.

In this paper, we focus on the nonsmooth composite optimization problems over networks, which consist of a smooth term and a nonsmooth term. Both equality constraints and box constraints for the decision variables are also considered. Based on the multi-agent networks, the objective problems are split into a series of agents on which the problems can be solved in a decentralized manner. By establishing the Lagrange function of the problems, the first-order optimal condition is obtained in the primal-dual domain. Then, we propose a decentralized algorithm with the proximal operators. The proposed algorithm has uncoordinated stepsizes with respect to agents or edges, where no global parameters are involved. By constructing the compact form of the algorithm with operators, we complete the convergence analysis with the fixed-point theory. With the constrained quadratic programming problem, simulations verify the effectiveness of the proposed algorithm.

Unified Robust Necessary Optimality Conditions for Nonconvex Nonsmooth Uncertain Multiobjective Optimization.

This paper is concerned with nonconvex nonsmooth uncertain multiobjective optimization problems, in which the decision variable of both objective and constraint functions is defined on Banach space while uncertain parameters are defined on arbitrary nonempty (may not be compact) sets. We employ the Stone–C̆ech compactification of uncertainty sets and the upper semicontinuous regularization of original functions with respect to uncertain parameters, giving rise to unified robust necessary optimality conditions for the local robust weakly efficient solution of the considered problem. Moreover, we derive weak and strong KKT robust necessary conditions via the constraint qualification and the regularity condition, respectively. Several examples are provided to illustrate the validity of our results.

On the exact l_{1} penalty function method for convex nonsmooth optimization problems with fuzzy objective function

In this paper, the convex nonsmooth optimization problem with fuzzy objective function and both inequality and equality constraints is considered. The Karush–Kuhn–Tucker necessary optimality conditions are proved for such a nonsmooth extremum problem. Further, the exact l_{1} penalty function method is used for solving the considered nonsmooth fuzzy optimization problem. Therefore, its associated fuzzy penalized optimization problem is constructed in this approach. Then, the exactness property of the exact l_{1} penalty function method is analyzed if it is used for solving the considered nonsmooth convex fuzzy optimization problem.