Nonsmooth Convex Minimization Research Articles

Novel algorithms based on forward-backward splitting technique: effective methods for regression and classification

In this paper, we introduce two novel forward-backward splitting algorithms (FBSAs) for nonsmooth convex minimization. We provide a thorough convergence analysis, emphasizing the new algorithms and contrasting them with existing ones. Our findings are validated through a numerical example. The practical utility of these algorithms in real-world applications, including machine learning for tasks such as classification, regression, and image deblurring reveal that these algorithms consistently approach optimal solutions with fewer iterations, highlighting their efficiency in real-world scenarios.

A Perturbation Framework for Convex Minimization and Monotone Inclusion Problems with Nonlinear Compositions

We introduce a framework based on Rockafellar’s perturbation theory to analyze and solve general nonsmooth convex minimization and monotone inclusion problems involving nonlinearly composed functions as well as linear compositions. Such problems have been investigated only from a primal perspective and only for nonlinear compositions of smooth functions in finite-dimensional spaces in the absence of linear compositions. In the context of Banach spaces, the proposed perturbation analysis serves as a foundation for the construction of a dual problem and of a maximally monotone Kuhn–Tucker operator, which is decomposable as the sum of simpler monotone operators. In the Hilbertian setting, this decomposition leads to a block-iterative primal-dual algorithm that fully splits all the components of the problem and appears to be the first proximal splitting algorithm for handling nonlinear composite problems. Various applications are discussed. Funding: The work of L. M. Briceño-Arias was supported by Agencia Nacional de Investigación y Desarrollo-Chile [Grant Fondo Nacional de Desarrollo Científico y Tecnológico 1190871, Grant Centro de Modelamiento Matemático ACE210010, Grant Centro de Modelamiento Matemático FB210005, and basal Funds for Centers of Excellence], and the work of P. L. Combettes was supported by the National Science Foundation [Grant DMS-1818946].

Efficient dual ADMMs for sparse compressive sensing MRI reconstruction

Magnetic Resonance Imaging (MRI) is a kind of medical imaging technology used for diagnostic imaging of diseases, but its image quality may be suffered by the long acquisition time. The compressive sensing (CS) based strategy may decrease the reconstruction time greatly, but it needs efficient reconstruction algorithms to produce high-quality and reliable images. This paper focuses on the algorithmic improvement for the sparse reconstruction of CS-MRI, especially considering a non-smooth convex minimization problem which is composed of the sum of a total variation regularization term and a $$\ell _1$$ -norm term of the wavelet transformation. The partly motivation of targeting the dual problem is that the dual variables are involved in relatively low-dimensional subspace. Instead of solving the primal model as usual, we turn our attention to its associated dual model composed of three variable blocks and two separable non-smooth function blocks. However, the directly extended alternating direction method of multipliers (ADMM) must be avoided because it may be divergent, although it usually performs well numerically. In order to solve the problem, we employ a symmetric Gauss–Seidel (sGS) technique based ADMM. Compared with the directly extended ADMM, this method only needs one additional iteration, but its convergence can be guaranteed theoretically. Besides, we also propose a generalized variant of ADMM because this method has been illustrated to be efficient for solving semidefinite programming in the past few years. Finally, we do extensive experiments on MRI reconstruction using some simulated and real MRI images under different sampling patterns and ratios. The numerical results demonstrate that the proposed algorithms significantly achieve high reconstruction accuracies with fast computational speed.

Vectorized MATLAB Implementation of the Incremental Minimization Principle for Rate-Independent Dissipative Solids Using FEM: A Constitutive Model of Shape Memory Alloys

The incremental energy minimization principle provides a compact variational formulation for evolutionary boundary problems based on constitutive models of rate-independent dissipative solids. In this work, we develop and implement a versatile computational tool for the resolution of these problems via the finite element method (FEM). The implementation is coded in the MATLAB programming language and benefits from vector operations, allowing all local energy contributions to be evaluated over all degrees of freedom at once. The monolithic solution scheme combined with gradient-based optimization methods is applied to the inherently nonlinear, non-smooth convex minimization problem. An advanced constitutive model for shape memory alloys, which features a strongly coupled rate-independent dissipation function and several constraints on internal variables, is implemented as a benchmark example. Numerical simulations demonstrate the capabilities of the computational tool, which is suited for the rapid development and testing of advanced constitutive laws of rate-independent dissipative solids.

Subgradient ellipsoid method for nonsmooth convex problems

In this paper, we present a new ellipsoid-type algorithm for solving nonsmooth problems with convex structure. Examples of such problems include nonsmooth convex minimization problems, convex-concave saddle-point problems and variational inequalities with monotone operator. Our algorithm can be seen as a combination of the standard Subgradient and Ellipsoid methods. However, in contrast to the latter one, the proposed method has a reasonable convergence rate even when the dimensionality of the problem is sufficiently large. For generating accuracy certificates in our algorithm, we propose an efficient technique, which ameliorates the previously known recipes (Nemirovski in Math Oper Res 35(1):52–78, 2010).

A New Homotopy Proximal Variable-Metric Framework for Composite Convex Minimization

This paper suggests two novel ideas to develop new proximal variable-metric methods for solving a class of composite convex optimization problems. The first idea is to utilize a new parameterization strategy of the optimality condition to design a class of homotopy proximal variable-metric algorithms that can achieve linear convergence and finite global iteration-complexity bounds. We identify at least three subclasses of convex problems in which our approach can apply to achieve linear convergence rates. The second idea is a new primal-dual-primal framework for implementing proximal Newton methods that has attractive computational features for a subclass of nonsmooth composite convex minimization problems. We specialize the proposed algorithm to solve a covariance estimation problem in order to demonstrate its computational advantages. Numerical experiments on the four concrete applications are given to illustrate the theoretical and computational advances of the new methods compared with other state-of-the-art algorithms.

A Robust Regularizer for Multiphase CT.

Joint image reconstruction for multiphase CT can potentially improve image quality and reduce dose by leveraging the shared information among the phases. Multiphase CT scans are acquired sequentially. Inter-scan patient breathing causes small organ shifts and organ boundary misalignment among different phases. Existing multi-channel regularizers such as the joint total variation (TV) can introduce artifacts at misaligned organ boundaries. We propose a multi-channel regularizer using the infimal convolution (inf-conv) between a joint TV and a separable TV. It is robust against organ misalignment; it can work like a joint TV or a separable TV depending on a parameter setting. The effects of the parameter in the inf-conv regularizer are analyzed in detail. The properties of the inf-conv regularizer are then investigated numerically in a multi-channel image denoising setting. For algorithm implementation, the inf-conv regularizer is nonsmooth; inverse problems with the inf-conv regularizer can be solved using a number of primal-dual algorithms from nonsmooth convex minimization. Our numerical studies using synthesized 2-phase patient data and phantom data demonstrate that the inf-conv regularizer can largely maintain the advantages of the joint TV over the separable TV and reduce image artifacts of the joint TV due to organ misalignment.

Convergence rate analysis of proximal gradient methods with applications to composite minimization problems

First-order methods such as proximal gradient, which use Forward–Backward Splitting techniques have proved to be very effective in solving nonsmooth convex minimization problem, which is useful in solving various practical problems in different fields such as machine learning and image processing. In this paper, we propose few new forward–backward splitting algorithms, which consume less number of iterations to converge to an optimum. In addition, we derive convergence rates for the proposed formulations and show that the speed of convergence of these algorithms is significantly better than the traditional forward–backward algorithm. To demonstrate the practical applicability, we apply them to two real-world problems of machine learning and image processing. The first issue deals with the regression on high-dimensional datasets, whereas the second one is the image deblurring problem. Numerical experiments have been conducted on several publicly available real datasets to verify the obtained theoretical results. Results demonstrate the superiority of our algorithms in terms of accuracy, the number of iterations required to converge and the rate of convergence against the classical first-order methods.

Convergence rate of a relaxed inertial proximal algorithm for convex minimization

ABSTRACTIn a Hilbert space setting, the authors recently introduced a general class of relaxed inertial proximal algorithms that aim to solve monotone inclusions. In this paper, we specialize this study in the case of non-smooth convex minimization problems. We obtain convergence rates for values which have similarities with the results based on the Nesterov accelerated gradient method. The joint adjustment of inertia, relaxation and proximal terms plays a central role. In doing so, we highlight inertial proximal algorithms that converge for general monotone inclusions, and which, in the case of convex minimization, give fast convergence rates of values in the worst case.

A family of inexact SQA methods for non-smooth convex minimization with provable convergence guarantees based on the Luo–Tseng error bound property

We propose a new family of inexact sequential quadratic approximation (SQA) methods, which we call the inexact regularized proximal Newton ($\textsf{IRPN}$) method, for minimizing the sum of two closed proper convex functions, one of which is smooth and the other is possibly non-smooth. Our proposed method features strong convergence guarantees even when applied to problems with degenerate solutions while allowing the inner minimization to be solved inexactly. Specifically, we prove that when the problem possesses the so-called Luo-Tseng error bound (EB) property, $\textsf{IRPN}$ converges globally to an optimal solution, and the local convergence rate of the sequence of iterates generated by $\textsf{IRPN}$ is linear, superlinear, or even quadratic, depending on the choice of parameters of the algorithm. Prior to this work, such EB property has been extensively used to establish the linear convergence of various first-order methods. However, to the best of our knowledge, this work is the first to use the Luo-Tseng EB property to establish the superlinear convergence of SQA-type methods for non-smooth convex minimization. As a consequence of our result, $\textsf{IRPN}$ is capable of solving regularized regression or classification problems under the high-dimensional setting with provable convergence guarantees. We compare our proposed $\textsf{IRPN}$ with several empirically efficient algorithms by applying them to the $\ell_1$-regularized logistic regression problem. Experiment results show the competitiveness of our proposed method.

Simulated Data for Linear Regression with Structured and Sparse Penalties: Introducing pylearn-simulate

A currently very active field of research is how to incorporate structure and prior knowledge in machine learning methods. It has lead to numerous developments in the field of non-smooth convex minimization. With recently developed methods it is possible to perform an analysis in which the computed model can be linked to a given structure of the data and simultaneously do variable selection to find a few important features in the data. However, there is still no way to unambiguously simulate data to test proposed algorithms, since the exact solutions to such problems are unknown. The main aim of this paper is to present a theoretical framework for generating simulated data. These simulated data are appropriate when comparing optimization algorithms in the context of linear regression problems with sparse and structured penalties. Additionally, this approach allows the user to control the signal-to-noise ratio, the correlation structure of the data and the optimization problem to which they are the solution. The traditional approach is to simulate random data without taking into account the actual model that will be fit to the data. But when using such an approach it is not possible to know the exact solution of the underlying optimization problem. With our contribution, it is possible to know the exact theoretical solution of a penalized linear regression problem, and it is thus possible to compare algorithms without the need to use, e.g., cross-validation. We also present our implementation, the Python package pylearn-simulate, available at https://github.com/neurospin/pylearn-simulate and released under the BSD 3clause license. We describe the package and give examples at the end of the paper.

A new accelerated proximal gradient technique for regularized multitask learning framework

Multitask learning can be defined as the joint learning of related tasks using shared representations, such that each task can help other tasks to perform better. One of the various multitask learning frameworks is the regularized convex minimization problem, for which many optimization techniques are available in the literature. In this paper, we consider solving the non-smooth convex minimization problem with sparsity-inducing regularizers for the multitask learning framework, which can be efficiently solved using proximal algorithms. Due to slow convergence of traditional proximal gradient methods, a recent trend is to introduce acceleration to these methods, which increases the speed of convergence. In this paper, we present a new accelerated gradient method for the multitask regression framework, which not only outperforms its non-accelerated counterpart and traditional accelerated proximal gradient method but also improves the prediction accuracy. We also prove the convergence and stability of the algorithm under few specific conditions. To demonstrate the applicability of our method, we performed experiments with several real multitask learning benchmark datasets. Empirical results exhibit that our method outperforms the previous methods in terms of convergence, accuracy and computational time.

Adaptive Proximal Average Approximation for Composite Convex Minimization

We propose a fast first-order method to solve multi-term nonsmooth composite convex minimization problems by employing a recent proximal average approximation technique and a novel adaptive parameter tuning technique. Thanks to this powerful parameter tuning technique, the proximal gradient step can be performed with a much larger stepsize in the algorithm implementation compared with the prior PA-APG method, which is the core to enable significant improvements in practical performance. Moreover, by choosing the approximation parameter adaptively, the proposed method is shown to enjoy the O(1/k) iteration complexity theoretically without needing any extra computational cost, while the PA-APG method incurs much more iterations for convergence. The preliminary experimental results on overlapping group Lasso and graph-guided fused Lasso problems confirm our theoretic claim well, and indicate that the proposed method is almost five times faster than the state-of-the-art PA-APG method and therefore suitable for higher-precision required optimization.

The Modified HZ Conjugate Gradient Algorithm for Large-Scale Nonsmooth Optimization.

In this paper, the Hager and Zhang (HZ) conjugate gradient (CG) method and the modified HZ (MHZ) CG method are presented for large-scale nonsmooth convex minimization. Under some mild conditions, convergent results of the proposed methods are established. Numerical results show that the presented methods can be better efficiency for large-scale nonsmooth problems, and several problems are tested (with the maximum dimensions to 100,000 variables).

String-averaging incremental subgradients for constrained convex optimization with applications to reconstruction of tomographic images

We present a method for non-smooth convex minimization which is based on subgradient directions and string-averaging techniques. In this approach, the set of available data is split into sequences (strings) and a given iterate is processed independently along each string, possibly in parallel, by an incremental subgradient method (ISM). The end-points of all strings are averaged to form the next iterate. The method is useful to solve sparse and large-scale non-smooth convex optimization problems, such as those arising in tomographic imaging. A convergence analysis is provided under realistic, standard conditions. Numerical tests are performed in a tomographic image reconstruction application, showing good performance for the convergence speed when measured as the decrease ratio of the objective function, in comparison to classical ISM.

Landweber-Kaczmarz method in Banach spaces with inexact inner solvers

In recent years the Landweber-Kaczmarz method has been proposed for solving nonlinear ill-posed inverse problems in Banach spaces using general convex penalty functions. The implementation of this method involves solving a (nonsmooth) convex minimization problem at each iteration step and the existing theory requires its exact resolution which in general is impossible in practical applications. In this paper we propose a version of the Landweber-Kaczmarz method in Banach spaces in which the minimization problem involved in each iteration step is solved inexactly. Based on the -subdifferential calculus we give a convergence analysis of our method. Furthermore, using Nesterov's strategy, we propose a possible accelerated version of the Landweber-Kaczmarz method. Numerical results on computed tomography and parameter identification in partial differential equations are provided to support our theoretical results and to demonstrate our accelerated method.

Decomposition Techniques for Bilinear Saddle Point Problems and Variational Inequalities with Affine Monotone Operators

The majority of first-order methods for large-scale convex---concave saddle point problems and variational inequalities with monotone operators are proximal algorithms. To make such an algorithm practical, the problem's domain should be proximal-friendly--admit a strongly convex function with easy to minimize linear perturbations. As a by-product, this domain admits a computationally cheap linear minimization oracle (LMO) capable to minimize linear forms. There are, however, important situations where a cheap LMO indeed is available, but the problem domain is not proximal-friendly, which motivates search for algorithms based solely on LMO. For smooth convex minimization, there exists a classical algorithm using LMO--conditional gradient. In contrast, known to us similar techniques for other problems with convex structure (nonsmooth convex minimization, convex---concave saddle point problems, even as simple as bilinear ones, and variational inequalities with monotone operators, even as simple as affine) are quite recent and utilize common approach based on Fenchel-type representations of the associated objectives/vector fields. The goal of this paper was to develop alternative (and seemingly much simpler) decomposition techniques based on LMO for bilinear saddle point problems and for variational inequalities with affine monotone operators.

A coordinate descent homotopy method for linearly constrained nonsmooth convex minimization

A problem in optimization, with a wide range of applications, entails finding a solution of a linear equation with various minimization properties. Such applications include compressed sensing, which requires an efficient method to find a minimal norm solution. We propose a coordinate descent homotopy method to solve the linearly constrained convex minimization problem where P is proper, convex and lower semicontinuous. A well-known special case is the basis pursuit problem . The greedy-type coordinate descent method is applied to solve the regularized linear least squares problem, which arises as a sequence of subproblems for the proposed method, and we show global linear convergence. We report numerical results for solving large-scale basis pursuit problem. Comparison with Bregman iterative algorithm [W. Yin, S. Osher, D. Goldfarb, and J. Darbon, Bregman iterative algorithms for -minimization with applications to compressed sensing, SIAM J. Image Sci. 1 (2008), pp. 143–168] and linearized Bregman iterative algorithm [J.-F. Cai, S. Osher, and Z. Shen, Linearized Bregman iterations for compressed sensing, Math. Comput. 78 (2009), pp. 1515–1536] suggests that the proposed method can be used as an efficient method for minimization problem.

On the rate of convergence of the proximal alternating linearized minimization algorithm for convex problems

We analyze the proximal alternating linearized minimization algorithm (PALM) for solving non-smooth convex minimization problems where the objective function is a sum of a smooth convex function and block separable non-smooth extended real-valued convex functions. We prove a global non-asymptotic sublinear rate of convergence for PALM. When the number of blocks is two, and the smooth coupling function is quadratic we present a fast version of PALM which is proven to share a global sublinear rate efficiency estimate improved by a squared root factor. Some numerical examples illustrate the potential benefits of the proposed schemes.

A modified PRP conjugate gradient algorithm with nonmonotone line search for nonsmooth convex optimization problems

It is well-known that nonlinear conjugate gradient (CG) methods are preferred to solve large-scale smooth optimization problems due to their simplicity and low storage. However, the CG methods for nonsmooth optimization have not been studied. In this paper, a modified Polak–Ribiere–Polyak CG algorithm which combines with a nonmonotone line search technique is proposed for nonsmooth convex minimization. The search direction of the given method not only possesses the sufficiently descent property but also belongs to a trust region. Moreover, the search direction has not only the gradients information but also the functions information. The global convergence of the presented algorithm is established under suitable conditions. Numerical results show that the given method is competitive to other three methods.