Nonsmooth Convex Optimization Problems Research Articles

Submodular functions: from discrete to continuous domains

Submodular set-functions have many applications in combinatorial optimization, as they can be minimized and approximately maximized in polynomial time. A key element in many of the algorithms and analyses is the possibility of extending the submodular set-function to a convex function, which opens up tools from convex optimization. Submodularity goes beyond set-functions and has naturally been considered for problems with multiple labels or for functions defined on continuous domains, where it corresponds essentially to cross second-derivatives being nonpositive. In this paper, we show that most results relating submodularity and convexity for set-functions can be extended to all submodular functions. In particular, (a) we naturally define a continuous extension in a set of probability measures, (b) show that the extension is convex if and only if the original function is submodular, (c) prove that the problem of minimizing a submodular function is equivalent to a typically non-smooth convex optimization problem, and (d) propose another convex optimization problem with better computational properties (e.g., a smooth dual problem). Most of these extensions from the set-function situation are obtained by drawing links with the theory of multi-marginal optimal transport, which provides also a new interpretation of existing results for set-functions. We then provide practical algorithms to minimize generic submodular functions on discrete domains, with associated convergence rates.

Local linear convergence analysis of Primal–Dual splitting methods

In this paper, we study the local linear convergence properties of a versatile class of Primal–Dual splitting methods for minimizing composite non-smooth convex optimization problems. Under the assumption that the non-smooth components of the problem are partly smooth relative to smooth manifolds, we present a unified local convergence analysis framework for these methods. More precisely, in our framework, we first show that (i) the sequences generated by Primal–Dual splitting methods identify a pair of primal and dual smooth manifolds in a finite number of iterations, and then (ii) enter a local linear convergence regime, which is characterized based on the structure of the underlying active smooth manifolds. We also show how our results for Primal–Dual splitting can be specialized to cover existing ones on Forward–Backward splitting and Douglas–Rachford splitting/ADMM (alternating direction methods of multipliers). Moreover, based on these obtained local convergence analysis result, several practical acceleration techniques are discussed. To exemplify the usefulness of the obtained result, we consider several concrete numerical experiments arising from fields including signal/image processing, inverse problems and machine learning. The demonstration not only verifies the local linear convergence behaviour of Primal–Dual splitting methods, but also the insights on how to accelerate them in practice.

Newton-type Alternating Minimization Algorithm for Convex Optimization

We propose NAMA (Newton-type Alternating Minimization Algorithm) for solving structured nonsmooth convex optimization problems where the sum of two functions is to be minimized, one being strongly convex and the other composed with a linear mapping. The proposed algorithm is a line-search method over a continuous, real-valued, exact penalty function for the corresponding dual problem, which is computed by evaluating the augmented Lagrangian at the primal points obtained by alternating minimizations. As a consequence, NAMA relies on exactly the same computations as the classical alternating minimization algorithm (AMA), also known as the dual proximal gradient method. Under standard assumptions the proposed algorithm possesses strong convergence properties, while under mild additional assumptions the asymptotic convergence is superlinear, provided that the search directions are chosen according to quasi-Newton formulas. Due to its simplicity, the proposed method is well suited for embedded applications and large-scale problems. Experiments show that using limited-memory directions in NAMA greatly improves the convergence speed over AMA and its accelerated variant.

Gradient-Free Two-Point Methods for Solving Stochastic Nonsmooth Convex Optimization Problems with Small Non-Random Noises

We study nonsmooth convex stochastic optimization problems with a two-point zero-order oracle, i.e., at each iteration one can observe the values of the function’s realization at two selected points. These problems are first smoothed out with the well-known technique of double smoothing (B.T. Polyak) and then solved with the stochastic mirror descent method. We obtain conditions for the permissible noise level of a nonrandom nature exhibited in the computation of the function’s realization for which the estimate on the method’s rate of convergence is preserved.

Accelerated proximal gradient method for elastoplastic analysis with von Mises yield criterion

It is known that, under certain conditions, the quasi-static incremental analysis problem of elastoplastic structures with the von Mises yield criterion can be formulated as a second-order cone programming (SOCP) problem, which can be solved with a primal-dual interior-point method. Alternatively, this paper proposes to solve an equivalent unconstrained nonsmooth convex optimization problem, which has a form similar to a class of regularized least-square problems, known as group LASSO. We propose an accelerated proximal gradient method with an adaptive restart scheme for solving this unconstrained optimization problem. The algorithm is easy to implement, and free from numerical solution of linear equations unlike conventional methods in computational mechanics. Numerical experiments suggest that the presented algorithm outperforms a standard solver that implements a primal-dual interior-point method for conic optimization.

A modified scaled memoryless BFGS preconditioned conjugate gradient algorithm for nonsmooth convex optimization

This paper presents a nonmonotone scaled memoryless BFGS preconditioned conjugate gradient algorithm for solving nonsmooth convex optimization problems, which combines the idea of scaled memoryless BFGS preconditioned conjugate gradient method with the nonmonotone technique and the Moreau-Yosida regularization. The proposed method makes use of approximate function and gradient values of the Moreau-Yosida regularization instead of the corresponding exact values. Under mild conditions, the global convergence of the proposed method is established. Preliminary numerical results and related comparisons show that the proposed method can be applied to solve large scale nonsmooth convex optimization problems.

Solving structured nonsmooth convex optimization with complexity mathcal {O}(varepsilon ^{-1/2})

This paper describes an algorithm for solving structured nonsmooth convex optimization problems using the optimal subgradient algorithm (OSGA), which is a first-order method with the complexity mathcal {O}(varepsilon ^{-2}) for Lipschitz continuous nonsmooth problems and mathcal {O}(varepsilon ^{-1/2}) for smooth problems with Lipschitz continuous gradient. If the nonsmoothness of the problem is manifested in a structured way, we reformulate the problem so that it can be solved efficiently by a new setup of OSGA (called OSGA-V) with the complexity mathcal {O}(varepsilon ^{-1/2}). Further, to solve the reformulated problem, we equip OSGA-O with an appropriate prox-function for which the OSGA-O subproblem can be solved either in a closed form or by a simple iterative scheme, which decreases the computational cost of applying the algorithm for large-scale problems. We show that applying the new scheme is feasible for many problems arising in applications. Some numerical results are reported confirming the theoretical foundations.

Recurrent networks with soft-thresholding nonlinearities for lightweight coding

A long-standing and influential hypothesis in neural information processing is that early sensory networks adapt themselves to produce efficient codes of afferent inputs. Here, we show how a nonlinear recurrent network provides an optimal solution for the efficient coding of an afferent input and its history. We specifically consider the problem of producing lightweight codes, ones that minimize both ℓ1 and ℓ2 constraints on sparsity and energy, respectively. When embedded in a linear coding paradigm, this problem results in a non-smooth convex optimization problem. We employ a proximal gradient descent technique to develop the solution, showing that the optimal code is realized through a recurrent network endowed with a nonlinear soft thresholding operator. The training of the network connection weights is readily achieved through gradient-based local learning. If such learning is assumed to occur on a slower time-scale than the (faster) recurrent dynamics, then the network as a whole converges to an optimal set of codes and weights via what is, in effect, an alternative minimization procedure. Our results show how the addition of thresholding nonlinearities to a recurrent network may enable the production of lightweight, history-sensitive encoding schemes.

CHI: A contemporaneous health index for degenerative disease monitoring using longitudinal measurements

In this paper, we develop a novel formulation for contemporaneous patient risk monitoring by exploiting the emerging data-rich environment in many healthcare applications, where an abundance of longitudinal data that reflect the degeneration of the health condition can be continuously collected. Our objective, and the developed formulation, is fundamentally different from many existing risk score models for different healthcare applications, which mostly focus on predicting the likelihood of a certain outcome at a pre-specified time. Rather, our formulation translates multivariate longitudinal measurements into a contemporaneous health index (CHI) that captures patient condition changes over the course of progression. Another significant feature of our formulation is that, CHI can be estimated with or without label information, different from other risk score models strictly based on supervised learning. To develop this formulation, we focus on the degenerative disease conditions, for which we could utilize the monotonic progression characteristic (either towards disease or recovery) to learn CHI. Such a domain knowledge leads us to a novel learning formulation, and on top of that, we further generalize this formulation with a capacity to incorporate label information if available. We further develop algorithms to mitigate the challenges associated with the nonsmooth convex optimization problem by first identifying its dual reformulation as a constrained smooth optimization problem, and then, using the block coordinate descent algorithm to iteratively solve the optimization with a derived efficient projection at each iteration. Extensive numerical studies are performed on both synthetic datasets and real-world applications on Alzheimer’s disease and Surgical Site Infection, which demonstrate the utility and efficacy of the proposed method on degenerative conditions that include a wide range of applications.

4-D light field reconstruction by irradiance decomposition

Common light sources such as an ordinary flashlight with lenses and/or reflectors make complex 4-D light field that cannot be represented by conventional isotropic distribution model nor point light source model. This paper describes a new approach to estimate 4-D light field using an illuminated diffuser. Unlike conventional works that capture a 4-D light field directly, our method decomposes observed intensities on the diffuser into intensities of 4-D light rays based on inverse rendering technique with prior knowledge. We formulate 4-D light field reconstruction problem as a non-smooth convex optimization problem for mathematically finding the global minimum.

A Modified Proximal Gradient Method for a Family of Nonsmooth Convex Optimization Problems

In this paper, we propose a modified proximal gradient method for solving a class of nonsmooth convex optimization problems, which arise in many contemporary statistical and signal processing applications. The proposed method adopts a new scheme to construct the descent direction based on the proximal gradient method. It is proven that the modified proximal gradient method is Q-linearly convergent without the assumption of the strong convexity of the objective function. Some numerical experiments have been conducted to evaluate the proposed method eventually.

Factorisable Multi-Task Quantile Regression

For many applications, analyzing multiple response variables jointly is desirable because of their dependency, and valuable information about the distribution can be retrieved by estimating quantiles. In this paper, we propose a multi-task quantile regression method that exploits the potential factor structure of multivariate conditional quantiles through nuclear norm regularization. We jointly study the theoretical properties and computational aspects of the estimating procedure. In particular, we develop an efficient iterative proximal gradient algorithm for the non-smooth and non-strictly convex optimization problem incurred in our estimating procedure, and derive oracle bounds for the estimation error in a realistic situation where the sample size and number of iterative steps are both finite. The finite iteration analysis is particular useful when the matrix to be estimated is big and the computational cost is high. Merits of the proposed methodology are demonstrated through a Monte Carlo experiment and applications to climatological and financial study. Specifically, our method provides an objective foundation for spatial extreme clustering, and gives a refreshing look on the global financial systemic risk. Supplementary materials for this article are available online.

Multi-Task Joint Sparse and Low-Rank Representation for the Scene Classification of High-Resolution Remote Sensing Image

Scene classification plays an important role in the intelligent processing of High-Resolution Satellite (HRS) remotely sensed images. In HRS image classification, multiple features, e.g., shape, color, and texture features, are employed to represent scenes from different perspectives. Accordingly, effective integration of multiple features always results in better performance compared to methods based on a single feature in the interpretation of HRS images. In this paper, we introduce a multi-task joint sparse and low-rank representation model to combine the strength of multiple features for HRS image interpretation. Specifically, a multi-task learning formulation is applied to simultaneously consider sparse and low-rank structures across multiple tasks. The proposed model is optimized as a non-smooth convex optimization problem using an accelerated proximal gradient method. Experiments on two public scene classification datasets demonstrate that the proposed method achieves remarkable performance and improves upon the state-of-art methods in respective applications.

Almost sure convergence of random projected proximal and subgradient algorithms for distributed nonsmooth convex optimization

Two distributed algorithms are described that enable all users connected over a network to cooperatively solve the problem of minimizing the sum of all users’ objective functions over the intersection of all users’ constraint sets, where each user has its own private nonsmooth convex objective function and closed convex constraint set, which is the intersection of a number of simple, closed convex sets. One algorithm enables each user to adjust its estimate using the proximity operator of its objective function and the metric projection onto one constraint set randomly selected from a number of simple, closed convex sets. The other determines each user’s estimate using the subdifferential of its objective function instead of the proximity operator. Investigation of the two algorithms’ convergence properties for a diminishing step-size rule revealed that, under certain assumptions, the sequences of all users generated by each of the two algorithms converge almost surely to the same solution. It also showed that the rate of convergence depends on the step size and that a smaller step size results in quicker convergence. The results of numerical evaluation using a nonsmooth convex optimization problem support the convergence analysis and demonstrate the effectiveness of the two algorithms.

Gradient-free proximal methods with inexact oracle for convex stochastic nonsmooth optimization problems on the simplex

In this paper we propose a modification of the mirror descent method for non-smooth stochastic convex optimization problems on the unit simplex. The optimization problems considered differ from the classical ones by availability of function values realizations. Our purpose is to derive the convergence rate of the method proposed and to determine the level of noise that does not significantly affect the convergence rate.

Finding and proving the exact ground state of a generalized Ising model by convex optimization and MAX-SAT

Lattice models, also known as generalized Ising models or cluster expansions, are widely used in many areas of science and are routinely applied to alloy thermodynamics, solid-solid phase transitions, magnetic and thermal properties of solids, and fluid mechanics, among others. However, the problem of finding the true global ground state of a lattice model, which is essential for all of the aforementioned applications, has remained unresolved, with only a limited number of results for highly simplified systems known. In this article, we present the first general algorithm to find the exact ground states of complex lattice models and to prove their global optimality, resolving this fundamental problem in condensed matter and materials theory. We transform the infinite-discrete-optimization problem into a pair of combinatorial optimization (MAX-SAT) and non-smooth convex optimization (MAX-MIN) problems, which provide upper and lower bounds on the ground state energy respectively. By systematically converging these bounds to each other, we find and prove the exact ground state of realistic Hamiltonians whose solutions are completely intractable via traditional methods. Considering that currently such Hamiltonians are solved using simulated annealing and genetic algorithms that are often unable to find the true global energy minimum, and never able to prove the optimality of their result, our work opens the door to resolving long-standing uncertainties in lattice models of physical phenomena.

String-averaging incremental subgradients for constrained convex optimization with applications to reconstruction of tomographic images

We present a method for non-smooth convex minimization which is based on subgradient directions and string-averaging techniques. In this approach, the set of available data is split into sequences (strings) and a given iterate is processed independently along each string, possibly in parallel, by an incremental subgradient method (ISM). The end-points of all strings are averaged to form the next iterate. The method is useful to solve sparse and large-scale non-smooth convex optimization problems, such as those arising in tomographic imaging. A convergence analysis is provided under realistic, standard conditions. Numerical tests are performed in a tomographic image reconstruction application, showing good performance for the convergence speed when measured as the decrease ratio of the objective function, in comparison to classical ISM.

Adaptive smoothing algorithms for nonsmooth composite convex minimization

We propose an adaptive smoothing algorithm based on Nesterov's smoothing technique in Nesterov (Math Prog 103(1):127---152, 2005) for solving fully nonsmooth composite convex optimization problems. Our method combines both Nesterov's accelerated proximal gradient scheme and a new homotopy strategy for smoothness parameter. By an appropriate choice of smoothing functions, we develop a new algorithm that has the $$\mathcal {O}\left( \frac{1}{\varepsilon }\right) $$O1?-worst-case iteration-complexity while preserves the same complexity-per-iteration as in Nesterov's method and allows one to automatically update the smoothness parameter at each iteration. Then, we customize our algorithm to solve four special cases that cover various applications. We also specify our algorithm to solve constrained convex optimization problems and show its convergence guarantee on a primal sequence of iterates. We demonstrate our algorithm through three numerical examples and compare it with other related algorithms.

A fast first-order optimization approach to elastoplastic analysis of skeletal structures

It is classical that, when the small deformation is assumed, the incremental analysis problem of an elastoplastic structure with a piecewise-linear yield condition and a linear strain hardening model can be formulated as a convex quadratic programming problem. Alternatively, this paper presents a different formulation, an unconstrained nonsmooth convex optimization problem, and proposes to solve it with an accelerated gradient-like method. Specifically, we adopt an accelerated proximal gradient method, that has been developed for a regularized least squares problem. Numerical experiments show that the presented algorithm is effective for large-scale elastoplastic analysis. Also, a simple warm-start strategy can speed up the algorithm when the path-dependent incremental analysis is carried out.

Kurtosis based weighted sparse model with convex optimization technique for bearing fault diagnosis

The bearing failure, generating harmful vibrations, is one of the most frequent reasons for machine breakdowns. Thus, performing bearing fault diagnosis is an essential procedure to improve the reliability of the mechanical system and reduce its operating expenses. Most of the previous studies focused on rolling bearing fault diagnosis could be categorized into two main families, kurtosis-based filter method and wavelet-based shrinkage method. Although tremendous progresses have been made, their effectiveness suffers from three potential drawbacks: firstly, fault information is often decomposed into proximal frequency bands and results in impulsive feature frequency band splitting (IFFBS) phenomenon, which significantly degrades the performance of capturing the optimal information band; secondly, noise energy spreads throughout all frequency bins and contaminates fault information in the information band, especially under the heavy noisy circumstance; thirdly, wavelet coefficients are shrunk equally to satisfy the sparsity constraints and most of the feature information energy are thus eliminated unreasonably. Therefore, exploiting two pieces of prior information (i.e., one is that the coefficient sequences of fault information in the wavelet basis is sparse, and the other is that the kurtosis of the envelope spectrum could evaluate accurately the information capacity of rolling bearing faults), a novel weighted sparse model and its corresponding framework for bearing fault diagnosis is proposed in this paper, coined KurWSD. KurWSD formulates the prior information into weighted sparse regularization terms and then obtains a nonsmooth convex optimization problem. The alternating direction method of multipliers (ADMM) is sequentially employed to solve this problem and the fault information is extracted through the estimated wavelet coefficients. Compared with state-of-the-art methods, KurWSD overcomes the three drawbacks and utilizes the advantages of both family tools. KurWSD has three main advantages: firstly, all the characteristic information scattered in proximal sub-bands is gathered through synthesizing those impulsive dominant sub-band signals and thus eliminates the dilemma of the IFFBS phenomenon. Secondly, the noises in the focused sub-bands could be alleviated efficiently through shrinking or removing the dense wavelet coefficients of Gaussian noise. Lastly, wavelet coefficients with faulty information are reliably detected and preserved through manipulating wavelet coefficients discriminatively based on the contribution to the impulsive components. Moreover, the reliability and effectiveness of the KurWSD are demonstrated through simulated and experimental signals.