Proximal Alternating Linearized Minimization Research Articles

A generalized inertial proximal alternating linearized minimization method for nonconvex nonsmooth problems

We propose a general inertial version of the proximal alternating linearized minimization (PALM) (denoted by NiPALM) for a class of nonconvex and nonsmooth minimization problems, whose objective function is the sum of a smooth function of the entire variables and two nonsmooth functions of each variable. NiPALM is general in the sense that it contains the popular PALM, the inertial PALM (iPALM) and Gauss-Seidel type inertial PALM (GiPALM) as special cases. Under mild assumptions, namely, the underlying functions satisfy the Kurdyka-Łojasiewicz (KL) property and some suitable conditions on the parameters, we prove that each bounded sequence generated by NiPALM globally converges to a critical point. We also apply NiPALM to nonnegative matrix factorization, sparse principal component analysis, and weighted low-rank matrix restoration problems. Comparing results with those by PALM, iPALM, and GiPALM demonstrate the robustness and effectiveness of the proposed algorithm.

Semi-Linearized Proximal Alternating Minimization for a Discrete Mumford-Shah Model.

The Mumford-Shah model is a standard model in image segmentation, and due to its difficulty, many approximations have been proposed. The major interest of this functional is to enable joint image restoration and contour detection. In this work, we propose a general formulation of the discrete counterpart of the Mumford-Shah functional, adapted to nonsmooth penalizations, fitting the assumptions required by the Proximal Alternating Linearized Minimization (PALM), with convergence guarantees. A second contribution aims to relax some assumptions on the involved functionals and derive a novel Semi-Linearized Proximal Alternated Minimization (SL-PAM) algorithm, with proved convergence. We compare the performances of the algorithm with several nonsmooth penalizations, for Gaussian and Poisson denoising, image restoration and RGB-color denoising. We compare the results with state-of-the-art convex relaxations of the Mumford-Shah functional, and a discrete version of the Ambrosio-Tortorelli functional. We show that the SL-PAM algorithm is faster than the original PALM algorithm, and leads to competitive denoising, restoration and segmentation results.

The Autoregressive Linear Mixture Model: A Time-Series Model for an Instantaneous Mixture of Network Processes

Vector autoregressive models provide a simple generative model for multivariate, time-series data. The autoregressive coefficients of the vector autoregressive model describe a network process. However, in real-world applications such as macroeconomics or neuroimaging, time-series data arise not from isolated network processes but instead from the simultaneous occurrence of multiple network processes. Standard vector autoregressive models cannot provide insights about the underlying structure of such time-series data. In this work, we present the autoregressive linear mixture (ALM) model. The ALM proposes a decomposition of time-series data into co-occurring network processes that we call autoregressive components. We also present a non-convex likelihood-based estimator for fitting the ALM model and show that it can be solved using the proximal alternating linearized minimization (PALM) algorithm. We validate the ALM on both synthetic and real-world electroencephalography data, showing that we can disambiguate task-relevant autoregressive components that correspond with distinct network processes.

Use of PALM for ℓ1 sparse matrix factorization: Difficulty and rationalization of a two-step approach

Blind Source Separation (BSS) is a key machine learning method, which has been successfully applied to analyze multichannel data in various domains ranging from medical imaging to astrophysics. Being an ill-posed matrix factorization problem, it is necessary to introduce extra regularizing priors on the sources. While using sparsity has led to improved factorization results, the quality of the separation process turns out to be dramatically dependent on the minimization strategy and the regularization parameters. In this scope, the Proximal Alternating Linearized Minimization (PALM) has recently attracted a lot of interest as a generic, fast and highly flexible algorithm. Using PALM for sparse BSS is theoretically well grounded, but getting good empirical results requires a fine tuning of the involved regularization parameters, which might be too computationally expensive with real-world large-scale data, therefore mandating automatic parameter choice strategies. In this article, we first investigate the empirical limitations of using the PALM algorithm to perform sparse BSS and we explain their origin. Based on this, we further study and justify an alternative two-step algorithmic framework combining PALM with a heuristic approach, namely the Generalized Morphological Component Analysis (GMCA). This method enables an automatic parameter choice for the PALM step. Numerical experiments with comparisons to standard algorithms are carried out on two realistic experiments in spectroscopy and astrophysics.

A Gauss–Seidel type inertial proximal alternating linearized minimization for a class of nonconvex optimization problems

In this paper we study a broad class of nonconvex and nonsmooth minimization problems, whose objective function is the sum of a smooth function of the entire variables and two nonsmooth functions of each variable. We adopt the framework of the proximal alternating linearized minimization (PALM), together with the inertial strategy to accelerate the convergence. Since the inertial step is performed once the x-subproblem/y-subproblem is updated, the algorithm is a Gauss–Seidel type inertial proximal alternating linearized minimization (GiPALM) algorithm. Under the assumption that the underlying functions satisfy the Kurdyka–Łojasiewicz (KL) property and some suitable conditions on the parameters, we prove that each bounded sequence generated by GiPALM globally converges to a critical point. We apply the algorithm to signal recovery, image denoising and nonnegative matrix factorization models, and compare it with PALM and the inertial proximal alternating linearized minimization.

Heuristics For Efficient Sparse Blind Source Separation

Sparse Blind Source Separation (sparse BSS) is a key method to analyze multichannel data in fields ranging from medical imaging to astrophysics. However, since it relies on seeking the solution of a non-convex penalized matrix factorization problem, its performances largely depend on the optimization strategy. In this context, Proximal Alternating Linearized Minimization (PALM) has become a standard algorithm which, despite its theoretical grounding, generally provides poor practical separation results. In this work, we first investigate the origins of these limitations, which are shown to take their roots in the sensitivity to both the initialization and the regularization parameter choice. As an alternative, we propose a novel strategy that combines a heuristic approach with PALM. We show its relevance on realistic astrophysical data.

Denoising Poisson phaseless measurements via orthogonal dictionary learning.

Phaseless diffraction measurements recorded by CCD detectors are often affected by Poisson noise. In this paper, we propose a dictionary learning model by employing patches based sparsity in order to denoise such Poisson phaseless measurements. The model consists of three terms: (i) A representation term by an orthogonal dictionary, (ii) an L0 pseudo norm of the coefficient matrix, and (iii) a Kullback-Leibler divergence term to fit phaseless Poisson data. Fast alternating minimization method (AMM) and proximal alternating linearized minimization (PALM) are adopted to solve the proposed model, and especially the theoretical guarantee of the convergence of PALM is provided. The subproblems for these two algorithms both have fast solvers, and indeed, the solutions for the sparse coding and dictionary updating both have closed forms due to the orthogonality of learned dictionaries. Numerical experiments for phase retrieval using coded diffraction and ptychographic patterns are conducted to show the efficiency and robustness of proposed methods, which, by preserving texture features, produce visually and quantitatively improved restored images compared with other phase retrieval algorithms without regularization and local sparsity promoting algorithms.

A penalty method for rank minimization problems in symmetric matrices

The problem of minimizing the rank of a symmetric positive semidefinite matrix subject to constraints can be cast equivalently as a semidefinite program with complementarity constraints (SDCMPCC). The formulation requires two positive semidefinite matrices to be complementary. This is a continuous and nonconvex reformulation of the rank minimization problem. We investigate calmness of locally optimal solutions to the SDCMPCC formulation and hence show that any locally optimal solution is a KKT point. We develop a penalty formulation of the problem. We present calmness results for locally optimal solutions to the penalty formulation. We also develop a proximal alternating linearized minimization (PALM) scheme for the penalty formulation, and investigate the incorporation of a momentum term into the algorithm. Computational results are presented.

Non-convex clustering via proximal alternating linearized minimization method

Clustering is a fundamental learning task in a wide range of research fields. The most popular clustering algorithm is arguably the K-means algorithm, it is well known that the performance of K-means algorithm heavily depends on initialization due to its strong non-convexity nature. To overcome the initialization issue, in this paper, we first relax the K-means model as an optimization problem with non-convex constraints, then employ the Proximal Alternating Linearized Minimization (PALM) method to solve the relaxed non-convex optimization model. The convergence analysis of PALM algorithm for the clustering problem is also provided. Experimental results on several benchmark datasets are conducted to evaluate the efficiency of our approach.

Inertial Proximal Alternating Linearized Minimization (iPALM) for Nonconvex and Nonsmooth Problems

In this paper we study nonconvex and nonsmooth optimization problems with semi-algebraic data, where the variables vector is split into several blocks of variables. The problem consists of one smooth function of the entire variables vector and the sum of nonsmooth functions for each block separately. We analyze an inertial version of the Proximal Alternating Linearized Minimization (PALM) algorithm and prove its global convergence to a critical point of the objective function at hand. We illustrate our theoretical findings by presenting numerical experiments on blind image deconvolution, on sparse non-negative matrix factorization and on dictionary learning, which demonstrate the viability and effectiveness of the proposed method.

A convergent least-squares regularized blind deconvolution approach

The aim of this work is to present a new and efficient optimization method for the solution of blind deconvolution problems with data corrupted by Gaussian noise, which can be reformulated as a constrained minimization problem whose unknowns are the point spread function (PSF) of the acquisition system and the true image. The objective function we consider is the weighted sum of the least-squares fit-to-data discrepancy and possible regularization terms accounting for specific features to be preserved in both the image and the PSF. The solution of the corresponding minimization problem is addressed by means of a proximal alternating linearized minimization (PALM) algorithm, in which the updating procedure is made up of one step of a gradient projection method along the arc and the choice of the parameter identifying the steplength in the descent direction is performed automatically by exploiting the optimality conditions of the problem. The resulting approach is a particular case of a general scheme whose convergence to stationary points of the constrained minimization problem has been recently proved. The effectiveness of the iterative method is validated in several numerical simulations in image reconstruction problems.

Proximal alternating linearized minimization for nonconvex and nonsmooth problems

We introduce a proximal alternating linearized minimization (PALM) algorithm for solving a broad class of nonconvex and nonsmooth minimization problems. Building on the powerful Kurdyka---?ojasiewicz property, we derive a self-contained convergence analysis framework and establish that each bounded sequence generated by PALM globally converges to a critical point. Our approach allows to analyze various classes of nonconvex-nonsmooth problems and related nonconvex proximal forward---backward algorithms with semi-algebraic problem's data, the later property being shared by many functions arising in a wide variety of fundamental applications. A by-product of our framework also shows that our results are new even in the convex setting. As an illustration of the results, we derive a new and simple globally convergent algorithm for solving the sparse nonnegative matrix factorization problem.