Nonsmooth Convex Optimization Problems Research Articles

Wei-Yao-Liu Conjugate Gradient Algorithm for Nonsmooth Convex Optimization Problems

This paper presents a Wei-Yao-Liu conjugate gradient algorithm for nonsmooth convex optimization problem. The proposed algorithm makes use of approximate function and gradient values of the Moreau-Yosida regularization function instead of the corresponding exact values. Under suitable conditions, the global convergence property could be established for the proposed conjugate gradient method. Finally, some numerical results are reported to show the efficiency of our algorithm.

A multi-step doubly stabilized bundle method for nonsmooth convex optimization

In this paper, by incorporating a multi-step scheme into the doubly stabilized bundle method (DSBM) recently developed by Oliveira and Solodov (2016), we propose a multi-step doubly stabilized bundle method (MDSBM) for solving nonsmooth convex optimization problems. In contrast to a single sequence generated by DSBM, the MDSBM generates three related iteration sequences. One is used to build the cutting-planes model of the objective function, another is served as the stability centers, and the third is the sequence of solutions to our new doubly stabilized subproblems. In addition, we present a new descent test criterion, aiming to take advantage of the multi-step scheme. We establish global convergence of the proposed method, and finally present some promising numerical results.

Non-stationary First-Order Primal-Dual Algorithms with Faster Convergence Rates

In this paper, we propose two novel non-stationary first-order primal-dual algorithms to solve non-smooth composite convex optimization problems. Unlike existing primal-dual schemes where the param...

Distributed Control Enforcing Group Sparsity in Smart Grids

In modern smart grids, charging of local energy storage devices is coordinated within the distribution grid to compensate the volatile aggregated power demand on the time interval of interest. However, this results in a perpetual usage of all batteries which in return reduces their lifetime. In this paper, we enforce group sparsity by using an ℓp,q-regularization on the control to counteract this phenomenon. This leads to a non-smooth convex optimization problem, for which a tailored Alternating Direction Method of Multipliers algorithm is proposed. Furthermore, the algorithm is embedded in a Model Predictive Control framework. Numerical simulations show that the proposed scheme yields sparse control while achieving reasonable overall peak shaving.

Random Minibatch Subgradient Algorithms for Convex Problems with Functional Constraints

In this paper we consider non-smooth convex optimization problems with (possibly) infinite intersection of constraints. In contrast to the classical approach, where the constraints are usually represented as intersection of simple sets, which are easy to project onto, in this paper we consider that each constraint set is given as the level set of a convex but not necessarily differentiable function. For these settings we propose subgradient iterative algorithms with random minibatch feasibility updates. At each iteration, our algorithms take a subgradient step aimed at only minimizing the objective function and then a subsequent subgradient step minimizing the feasibility violation of the observed minibatch of constraints. The feasibility updates are performed based on either parallel or sequential random observations of several constraint components. We analyze the convergence behavior of the proposed algorithms for the case when the objective function is strongly convex and with bounded subgradients, while the functional constraints are endowed with a bounded first-order black-box oracle. For a diminishing stepsize, we prove sublinear convergence rates for the expected distances of the weighted averages of the iterates from the constraint set, as well as for the expected suboptimality of the function values along the weighted averages. Our convergence rates are known to be optimal for subgradient methods on this class of problems. Moreover, the rates depend explicitly on the minibatch size and show when minibatching helps a subgradient scheme with random feasibility updates.

Incremental and Parallel Machine Learning Algorithms With Automated Learning Rate Adjustments.

The existing machine learning algorithms for minimizing the convex function over a closed convex set suffer from slow convergence because their learning rates must be determined before running them. This paper proposes two machine learning algorithms incorporating the line search method, which automatically and algorithmically finds appropriate learning rates at run-time. One algorithm is based on the incremental subgradient algorithm, which sequentially and cyclically uses each of the parts of the objective function; the other is based on the parallel subgradient algorithm, which uses parts independently in parallel. These algorithms can be applied to constrained nonsmooth convex optimization problems appearing in tasks of learning support vector machines without adjusting the learning rates precisely. The proposed line search method can determine learning rates to satisfy weaker conditions than the ones used in the existing machine learning algorithms. This implies that the two algorithms are generalizations of the existing incremental and parallel subgradient algorithms for solving constrained nonsmooth convex optimization problems. We show that they generate sequences that converge to a solution of the constrained nonsmooth convex optimization problem under certain conditions. The main contribution of this paper is the provision of three kinds of experiment showing that the two algorithms can solve concrete experimental problems faster than the existing algorithms. First, we show that the proposed algorithms have performance advantages over the existing ones in solving a test problem. Second, we compare the proposed algorithms with a different algorithm Pegasos, which is designed to learn with a support vector machine efficiently, in terms of prediction accuracy, value of the objective function, and computational time. Finally, we use one of our algorithms to train a multilayer neural network and discuss its applicability to deep learning.

A Nonmonotone Hestenes ans Stiefel Conjugate Gradient Algorithm for Nonsmooth Convex Optimization

Here, we propose a practical method for solving nonsmooth convex problems by using conjugate gradient type methods. We present a modified HS conjugate gradient method, as one of the most remarkable methods to solve smooth and large-scale optimization problems. In the case that we have a nonsmooth convex problem, by way of the Moreau-Yosida regularization, we convert the nonsmooth objective function to a smooth function and then we use our method, by making use of a nonmonotone line search, for solving a nonsmooth convex optimization problem. We prove that our algorithm converges to an optimal solution under standard condition. Our algorithm inherits the performance of HS conjugate gradient method.

Asynchronous level bundle methods

In this paper, we consider nonsmooth convex optimization problems with additive structure featuring independent oracles (black-boxes) working in parallel. Existing methods for solving these distributed problems in a general form are synchronous, in the sense that they wait for the responses of all the oracles before performing a new iteration. In this paper, we propose level bundle methods handling asynchronous oracles. These methods require original upper-bounds (using upper-models or scarce coordinations) to deal with asynchronicity. We prove their convergence using variational-analysis techniques and illustrate their practical performance on a Lagrangian decomposition problem.

Total roto-translational variation

We consider curvature depending variational models for image regularization, such as Euler's elastica. These models are known to provide strong priors for the continuity of edges and hence have important applications in shape-and image processing. We consider a lifted convex representation of these models in the roto-translation space: In this space, curvature depending variational energies are represented by means of a convex functional defined on divergence free vector fields. The line energies are then easily extended to any scalar function. It yields a natural generalization of the total variation to the roto-translation space. As our main result, we show that the proposed convex representation is tight for characteristic functions of smooth shapes. We also discuss cases where this representation fails. For numerical solution, we propose a staggered grid discretization based on an averaged Raviart-Thomas finite elements approximation. This discretization is consistent, up to minor details, with the underlying continuous model. The resulting non-smooth convex optimization problem is solved using a first-order primal-dual algorithm. We illustrate the results of our numerical algorithm on various problems from shape-and image processing.

Group sparse regularization for impact force identification in time domain

Impact force identification remains a challenging inverse problem, where a small error in structural responses may lead to a large deviation in the actual solution. Recently, sparse regularization has attracted a lot of interest in the field of force identification. However, the standard sparse regularization method for force identification does not consider the intrinsic structure of impact force, i.e., the nonzero elements occur in groups, called group sparsity. In this paper, by exploiting the group sparse structure of impact force time history, we develop a general group sparse regularization method based on minimizing mixed l2,1-norm for the inverse problem of impact force identification. First, the number and size of groups including a complete profile of impact force are determined for penalizing the sum of the l2-norm of groups associated with the pulse profile of impact force. Second, a general group sparse optimization model based on the mixed l2,1-norm penalty for impact force identification is constructed in time domain, leading to a non-smooth convex optimization problem. Third, given the transfer function and the impact response, an accelerated gradient descent method is developed to solve such a group sparse regularization model. Finally, experiments including identification of single and consecutive impact forces are conducted on a clamped-free thin plate to illustrate the effectiveness and applicability of the proposed approach. Experimental results demonstrate that the classical Tikhonov regularization methods can only identify the single impact force from weakly noisy responses; the group sparse regularization method can efficiently identify both single and consecutive impact forces from heavily noisy responses, and has a slightly better improvement of the peak force amplitude than the standard sparse regularization method.

The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization.

The extrapolation strategy raised by Nesterov, which can accelerate the convergence rate of gradient descent methods by orders of magnitude when dealing with smooth convex objective, has led to tremendous success in training machine learning tasks. In this article, the convergence of individual iterates of projected subgradient (PSG) methods for nonsmooth convex optimization problems is theoretically studied based on Nesterov's extrapolation, which we name individual convergence. We prove that Nesterov's extrapolation has the strength to make the individual convergence of PSG optimal for nonsmooth problems. In light of this consideration, a direct modification of the subgradient evaluation suffices to achieve optimal individual convergence for strongly convex problems, which can be regarded as making an interesting step toward the open question about stochastic gradient descent (SGD) posed by Shamir. Furthermore, we give an extension of the derived algorithms to solve regularized learning tasks with nonsmooth losses in stochastic settings. Compared with other state-of-the-art nonsmooth methods, the derived algorithms can serve as an alternative to the basic SGD especially in coping with machine learning problems, where an individual output is needed to guarantee the regularization structure while keeping an optimal rate of convergence. Typically, our method is applicable as an efficient tool for solving large-scale l1 -regularized hinge-loss learning problems. Several comparison experiments demonstrate that our individual output not only achieves an optimal convergence rate but also guarantees better sparsity than the averaged solution.

A symmetric Gauss-Seidel based method for a class of multi-period mean-variance portfolio selection problems

It is commonly accepted that the estimation error of asset returns' sample mean is much larger than that of sample covariance. In order to hedge the risk raised by the estimation error of the sample mean, we propose a sparse and robust multi-period mean-variance portfolio selection model and show how this proposed model can be equivalently reformulated as a multi-block nonsmooth convex optimization problem. In order to get an optimal strategy, a symmetric Gauss-Seidel based method is implemented. Moreover, we show that the algorithm is globally linearly convergent. The effectiveness of our portfolio selection model and the efficiency of its solution method are demonstrated by empirical experiments on both the synthetic and real datasets.

Distributed quasi-monotone subgradient algorithm for nonsmooth convex optimization over directed graphs

Distributed optimization is of essential importance in networked systems. Most of the existing distributed algorithms either assume the information exchange over undirected graphs, or require that the underlying directed network topology provides a doubly stochastic weight matrix to the agents. In this brief paper, a distributed subgradient-based algorithm is proposed to solve nonsmooth convex optimization problems. The algorithm applies to directed graphs without using a doubly stochastic weight matrix. Moreover, the algorithm is a distributed generalization and improvement of the quasi-monotone subgradient algorithm. An O(1∕k) convergence rate is achieved. The effectiveness of our algorithm is also illustrated by a numerical example.

Projected Primal-Dual Dynamics for Distributed Constrained Nonsmooth Convex Optimization.

A distributed nonsmooth convex optimization problem subject to a general type of constraint, including equality and inequality as well as bounded constraints, is studied in this paper for a multiagent network with a fixed and connected communication topology. To collectively solve such a complex optimization problem, primal-dual dynamics with projection operation are investigated under optimal conditions. For the nonsmooth convex optimization problem, a framework under the LaSalle's invariance principle from nonsmooth analysis is established, where the asymptotic stability of the primal-dual dynamics at an optimal solution is guaranteed. For the case where inequality and bounded constraints are not involved and the objective function is twice differentiable and strongly convex, the globally exponential convergence of the primal-dual dynamics is established. Finally, two simulations are provided to verify and visualize the theoretical results.

Distributed Coordination for Nonsmooth Convex Optimization via Saddle-Point Dynamics

This paper considers continuous-time coordination algorithms for networks of agents that seek to collectively solve a general class of nonsmooth convex optimization problems with an inherent distributed structure. Our algorithm design builds on the characterization of the solutions of the nonsmooth convex program as saddle points of an augmented Lagrangian. We show that the associated saddle-point dynamics are asymptotically correct but, in general, not distributed because of the presence of a global penalty parameter. This motivates the design of a discontinuous saddle-point-like algorithm that enjoys the same convergence properties and is fully amenable to distributed implementation. Our convergence proofs rely on the identification of a novel global Lyapunov function for saddle-point dynamics. This novelty also allows us to identify mild convexity and regularity conditions on the objective function that guarantee the exponential convergence rate of the proposed algorithms for convex optimization problems subject to equality constraints. Various examples illustrate our discussion.

Multi-task image set classification via joint representation with class-level sparsity and intra-task low-rankness

Image set classification has recently attracted great attention due to its widespread applications in computer vision and pattern recognition. The great challenges lie in effectively and efficiently measuring the similarity among image sets with high inter-class ambiguity and large intra-class variability. In this paper, we propose a joint representation based approach to image set classification, in which class-level sparse and globally low rank constraints are imposed on the representation coefficients to embody inter-set discrimination and intra-set commonality respectively. Furthermore, sometimes the small size of image sets or improper usage of a single kind of features causes useful information limited and lacking in discriminability. To address this problem, we extend the traditional image set classification to a multi-task version, i.e., modify the proposed model to incorporate multiple kinds of features. Fortunately, on the total multi-task representation coefficients, both the total class-level sparsity and the intra-task low-rankness constraints still apply. The proposed method is optimized as a non-smooth convex optimization problem by employing an alternating optimization technique. Experiments on five public datasets demonstrate that the proposed method surpasses existing joint representation models with various regularizations for image set classification and compares favorably with other state-of-the-art methods.

A universal modification of the linear coupling method

ABSTRACTIn the late sixties, N. Shor and B. Polyak independently proposed optimal first-order methods for solving non-smooth convex optimization problems. In 1982 A. Nemirovski proposed optimal first-order methods for solving smooth convex optimization problems, which utilized auxiliary line search. In 1985 A. Nemirovski and Yu. Nesterov proposed a parametric family of optimal first-order methods for solving convex optimization problems with intermediate smoothness. In 2013 Yu. Nesterov proposed a universal gradient method which combined all good properties of the previous methods, except the possibility of using auxiliary line search. One can typically observe that in practice auxiliary line search improves performance for many tasks. In this paper, we propose the apparently first such method of non-smooth convex optimization allowing the use of the line search procedure. Moreover, it is based on the universal gradient method, which does not require any a priori information about the actual degree of smoothness of the problem. Numerical experiments demonstrate that the proposed method is, in some cases, considerably faster than Nesterov's universal gradient method.

Gradient-Free Two-Point Methods for Solving Stochastic Nonsmooth Convex Optimization Problems with Small Non-Random Noises

Gradient-Free Two-Point Methods for Solving Stochastic Nonsmooth Convex Optimization Problems with Small Non-Random Noises

A Distributed ADMM Approach With Decomposition-Coordination for Mobile Data Offloading

Mobile data offloading allows alternative network systems such as Wi-Fi hotspot access points (APs) to offload the traffic that is originally targeted for the cellular network operators (CNOs). It can alleviate the cellular network congestions and enhance users' quality-of-service. One of the main challenges of the mobile data offloading is to develop simple and distributed traffic allocation strategies that can optimally allocate the cellular data traffic from multiple CNOs to the APs. In this work, we propose a distributed optimization framework with decomposition-coordination to solve this problem. In our framework, the utility maximization problem of the cellular network system with mobile data offloading is formulated as a nonsmooth convex optimization problem. We then divide this problem into a set of subproblems, and each of them is solved by a CNO or an AP using its local information. All the subproblems are coordinated with each other by a virtual data offloading coordinator (VDOC), which can collect the intermediate calculation values from CNOs and APs. The VDOC will then feedback the coordination results calculated from the collected values to the corresponding CNOs and APs. We propose two distributed algorithms that can achieve the global optimization solution under different scenarios. The first one is the multiblock proximal Jacobi alternating direction method of multipliers (ProxJ-ADMM). In this algorithm, the communication between the VDOC and CNOs or APs is assumed to be fully synchronized, and the VDOC will only feedback the coordination result after successfully receiving all the values from APs and CNOs. We relax this assumption by introducing the second algorithm referred to as the distributed asynchronized ADMM (Async-ADMM) algorithm. In this algorithm, the coordination between the VDOC and APs or CNOs does not need to be perfectly synchronized and the VDOC will feedback a coordination result whenever it receives a value from at least one AP or CNO. We prove that both algorithm can achieve the global optimal solution. We present the numerical results to verify the performance of our proposed approaches under various network settings and conditions.

Proximal gradient method with automatic selection of the parameter by automatic differentiation

A class of non-smooth convex optimization problems which arise naturally from applications in sparse group Lasso, have attracted significant research efforts for parameters selection. For given parameters of the problem, proximal gradient method (PGM) is effective to solve it with linear convergence rate and the closed form solution can be obtained at each iteration. However, in many practical applications, the selection of the parameters not only affects the quality of solution, but also even determines whether the solution is right or not. In this paper, we study a new method to analyse the impact of the parameters on PGM algorithm to solve the non-smooth convex optimization problem. We present the sensitivity analysis on the output of an optimization algorithm over parameter, and show the advantage of the technique using automatic differentiation. Then, we propose a hybrid algorithm for selecting the optimal parameter based on the method of PGM. The numerical results show that the proposed method is effective for the solving of sparse signal recovery problem.