Proximal Gradient Algorithm Research Articles

A difference-of-convex approach for split feasibility with applications to matrix factorizations and outlier detection

The split feasibility problem is to find an element in the intersection of a closed set $C$ and the linear preimage of another closed set $D$, assuming the projections onto $C$ and $D$ are easy to compute. This class of problems arises naturally in many contemporary applications such as compressed sensing. While the sets $C$ and $D$ are typically assumed to be convex in the literature, in this paper, we allow both sets to be possibly nonconvex. We observe that, in this setting, the split feasibility problem can be formulated as an optimization problem with a difference-of-convex objective so that standard majorization-minimization type algorithms can be applied. Here we focus on the nonmonotone proximal gradient algorithm with majorization studied in [15, Appendix A]. We show that, when this algorithm is applied to a split feasibility problem, the sequence generated clusters at a stationary point of the problem under mild assumptions. We also study local convergence property of the sequence under suitable assumptions on the closed sets involved. Finally, we perform numerical experiments to illustrate the efficiency of our approach on solving split feasibility problems that arise in completely positive matrix factorization, (uniformly) sparse matrix factorization, and outlier detection.

Superiorization methodology and perturbation resilience of inertial proximal gradient algorithm with application to signal recovery

In this paper, we construct a novel algorithm for solving non-smooth composite optimization problems. By using inertial technique, we propose a modified proximal gradient algorithm with outer perturbations, and under standard mild conditions, we obtain strong convergence results for finding a solution of composite optimization problem. Based on bounded perturbation resilience, we present our proposed algorithm with the superiorization method and apply it to image recovery problem. Finally, we provide the numerical experiments to show efficiency of the proposed algorithm and comparison with previously known algorithms in signal recovery.

Penalised robust estimators for sparse and high-dimensional linear models

We introduce a new class of robust M-estimators for performing simultaneous parameter estimation and variable selection in high-dimensional regression models. We first explain the motivations for the key ingredient of our procedures which are inspired by regularization methods used in wavelet thresholding in noisy signal processing. The derived penalized estimation procedures are shown to enjoy theoretically the oracle property both in the classical finite dimensional case as well as the high-dimensional case when the number of variables p is not fixed but can grow with the sample size n, and to achieve optimal asymptotic rates of convergence. A fast accelerated proximal gradient algorithm, of coordinate descent type, is proposed and implemented for computing the estimates and appears to be surprisingly efficient in solving the corresponding regularization problems including the case for ultra high-dimensional data where $$p \gg n$$ . Finally, a very extensive simulation study and some real data analysis, compare several recent existing M-estimation procedures with the ones proposed in the paper, and demonstrate their utility and their advantages.

Bregman Proximal Mappings and Bregman–Moreau Envelopes Under Relative Prox-Regularity

We systematically study the local single-valuedness of the Bregman proximal mapping and local smoothness of the Bregman–Moreau envelope of a nonconvex function under relative prox-regularity—an extension of prox-regularity—which was originally introduced by Poliquin and Rockafellar. As Bregman distances are asymmetric in general, in accordance with Bauschke et al., it is natural to consider two variants of the Bregman proximal mapping, which, depending on the order of the arguments, are called left and right Bregman proximal mapping. We consider the left Bregman proximal mapping first. Then, via translation result, we obtain analogue (and partially sharp) results for the right Bregman proximal mapping. The class of relatively prox-regular functions significantly extends the recently considered class of relatively hypoconvex functions. In particular, relative prox-regularity allows for functions with a possibly nonconvex domain. Moreover, as a main source of examples and analogously to the classical setting, we introduce relatively amenable functions, i.e. convexly composite functions, for which the inner nonlinear mapping is component-wise smooth adaptable, a recently introduced extension of Lipschitz differentiability. By way of example, we apply our theory to locally interpret joint alternating Bregman minimization with proximal regularization as a Bregman proximal gradient algorithm, applied to a smooth adaptable function.

Forecasting the real prices of crude oil using robust regression models with regularization constraints

In this paper, we forecast the real price of crude oil via a robust loss function (Huber), with regularization constraints including LASSO, Ridge, and Elastic Net. These modifications are designed to avoid problems with overfitting and improve out-of-sample predictive performance. The efficient implementation of penalized regression for Huber losses is supported by the accelerated proximal gradient algorithm. Our results indicate that equal-weight mean combinations based on robust parameter design and parameterization penalties can outperform the benchmark no-change model at all horizons (up to two years). We also find that combinations of forecasts from robust penalized models can significantly outperform those based on OLS in horizons of longer than three months. These models have consistent and significantly higher directional accuracy than the no-change model, with success ratios of up to 63.9%.

Normalization of face illumination with photorealistic texture via deep image prior synthesis

Illumination normalization for face recognition is of great importance when the face image is taken under a harsh lighting condition and only one single image is available. In this paper, an illumination normalization method that can preserve face identity and make the processed image have photorealistic face texture is proposed. First, an illumination regression filter is proposed to remove a large number of illumination components of the input image. Then, the accelerated proximal gradient algorithm is used for low-rank decomposition of the filtered image, which further reduces the residual illumination and noise of the face image. However, after these two operations, the face image has a peeled-off appearance. To avoid the unrealistic appearance, a deep image prior synthesis process is developed to synthesize the photorealistic face texture. To accomplish this task, the filtered image, the low-rank decomposed image, the original image and any a uniformly illuminated frontal face image as reference are used as synthetic ingredients. These ingredients, as prior knowledge, synthesize a face image that has the texture features of the reference image and retains the identity of the input image. We try to bridge the appearance gap between the synthetic ingredients and the final synthetic image by a deep neural network. Instead of training the network on a large dataset, the final synthetic image is generated by iterating the parameters of the generator-structured network according to a certain proportion of the synthetic ingredients. Systematic evaluations conducted on public databases demonstrate that the proposed method is robust to illumination with better performance than state-of-the-art methods.

Convex-Concave Backtracking for Inertial Bregman Proximal Gradient Algorithms in Nonconvex Optimization

Backtracking line-search is an old yet powerful strategy for finding better step sizes to be used in proximal gradient algorithms. The main principle is to locally find a simple convex upper bound ...

Newton-like Inertial Dynamics and Proximal Algorithms Governed by Maximally Monotone Operators

The introduction of the Hessian damping in the continuous version of Nesterov's accelerated gradient method provides, by temporal discretization, fast proximal gradient algorithms where the oscillations are significantly attenuated. We will extend these results to the maximally monotone case. We rely on the technique introduced by Attouch and Peypouquet [Math. Program., 174 (2019), pp. 391--432], where the maximally monotone operator is replaced by its Yosida approximation with an appropriate adjustment of the regularization parameter. In a general Hilbert framework, we obtain the weak convergence of the iterates to equilibria, and the rapid convergence of the discrete velocities to zero. By specializing these algorithms to convex minimization, we obtain the convergence rate $o(1/k^2)$ of the values, and the rapid convergence of the gradients toward zero.

A Distributed Flexible Delay-Tolerant Proximal Gradient Algorithm

We develop and analyze an asynchronous algorithm for distributed convex optimization when the objective can be written as a sum of smooth functions, local to each worker, and a nonsmooth function. Unlike many existing methods, our distributed algorithm is adjustable to various levels of communication cost, delays, machines' computational power, and functions' smoothness. A unique feature is that the step sizes do not depend on communication delays nor number of machines, which is highly desirable for scalability. We prove that the algorithm converges linearly in the strongly convex case, and provide guarantees of convergence for the non-strongly convex case. The obtained rates are the same as the vanilla proximal gradient algorithm over some introduced epoch sequence that subsumes the delays of the system. We provide numerical results on large-scale machine learning problems to demonstrate the merits of the proposed method.

Maximum Likelihood Estimation of Regularization Parameters in High-Dimensional Inverse Problems: An Empirical Bayesian Approach. Part II: Theoretical Analysis

This paper presents a detailed theoretical analysis of the three stochastic approximation proximal gradient algorithms proposed in our companion paper [A. F. Vidal et al., SIAM J. Imaging Sci., 13 ...

Generalized Convolutional Sparse Coding with Unknown Noise.

Convolutional sparse coding (CSC) can learn representative shift-invariant patterns from multiple kinds of data. However, existing CSC methods can only model noises from Gaussian distribution, which is restrictive and unrealistic. In this paper, we propose a generalized CSC model capable of dealing with complicated unknown noise. The noise is now modeled by Gaussian mixture model, which can approximate any continuous probability density function. We use the expectation-maximization algorithm to solve the problem and design an efficient method for the weighted CSC problem in maximization step. The crux is to speed up the convolution in the frequency domain while keeping the other computations involving weight matrix in the spatial domain. Besides, we simultaneously update the dictionary and codes by nonconvex accelerated proximal gradient algorithm without bringing in extra alternating loops. The resultant method, called generalized convolutional sparse coding (GCSC), obtains the same space complexity and a smaller running time compared with existing CSC methods. Extensive experiments on synthetic and real-world noisy data sets validate that GCSC can model noise effectively and obtain high-quality filters and representations.

Finite Convergence of Proximal-Gradient Inertial Algorithms Combining Dry Friction with Hessian-Driven Damping

In a Hilbert space ${\mathcal H}$, we introduce a new class of proximal-gradient algorithms with finite convergence properties. These algorithms naturally occur as discrete temporal versions of an ...

Convergence of Stochastic Proximal Gradient Algorithm

We study the extension of the proximal gradient algorithm where only a stochastic gradient estimate is available and a relaxation step is allowed. We establish convergence rates for function values in the convex case, as well as almost sure convergence and convergence rates for the iterates under further convexity assumptions. Our analysis avoid averaging the iterates and error summability assumptions which might not be satisfied in applications, e.g. in machine learning. Our proofing technique extends classical ideas from the analysis of deterministic proximal gradient algorithms.

A Decentralized Proximal-Gradient Method With Network Independent Step-Sizes and Separated Convergence Rates

This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a previous algorithm, PG-EXTRA \cite{shi2015proximal}, but has a few advantages. First of all, agents use uncoordinated step-sizes, and the stable upper bounds on step-sizes are independent of network topologies. The step-sizes depend on local objective functions, and they can be as large as those of the gradient descent. Secondly, for the special case without non-smooth terms, linear convergence can be achieved under the strong convexity assumption. The dependence of the convergence rate on the objective functions and the network are separated, and the convergence rate of the new algorithm is as good as one of the two convergence rates that match the typical rates for the general gradient descent and the consensus averaging. We provide numerical experiments to demonstrate the efficacy of the introduced algorithm and validate our theoretical discoveries.

Accelerated iterative hard thresholding algorithm for $$l_0$$ regularized regression problem

In this paper, we propose an accelerated iterative hard thresholding algorithm for solving the $$l_0$$ regularized box constrained regression problem. We substantiate that there exists a threshold, if the extrapolation coefficients are chosen below this threshold, the proposed algorithm is equivalent to the accelerated proximal gradient algorithm for solving a corresponding constrained convex problem after finite iterations. Under some proper conditions, we get that the sequence generated by the proposed algorithm is convergent to a local minimizer of the $$l_0$$ regularized problem, which satisfies a desired lower bound. Moreover, when the data fitting function satisfies the error bound condition, we prove that both the iterate sequence and the corresponding sequence of objective function values are R-linearly convergent. Finally, we use several numerical experiments to verify our theoretical results.

Transformed [formula omitted] regularization for learning sparse deep neural networks

Deep Neural Networks (DNNs) have achieved extraordinary success in numerous areas. However, DNNs often carry a large number of weight parameters, leading to the challenge of heavy memory and computation costs. Overfitting is another challenge for DNNs when the training data are insufficient. These challenges severely hinder the application of DNNs in resource-constrained platforms. In fact, many network weights are redundant and can be removed from the network without much loss of performance. In this paper, we introduce a new non-convex integrated transformed ℓ1 regularizer to promote sparsity for DNNs, which removes redundant connections and unnecessary neurons simultaneously. Specifically, we apply the transformed ℓ1 regularizer to the matrix space of network weights and utilize it to remove redundant connections. Besides, group sparsity is integrated to remove unnecessary neurons. An efficient stochastic proximal gradient algorithm is presented to solve the new model. To the best of our knowledge, this is the first work to develop a non-convex regularizer in sparse optimization based method to simultaneously promote connection-level and neuron-level sparsity for DNNs. Experiments on public datasets demonstrate the effectiveness of the proposed method.

A Probabilistic Incremental Proximal Gradient Method

In this letter, we propose a probabilistic optimization method, named probabilistic incremental proximal gradient (PIPG) method, by developing a probabilistic interpretation of the incremental proximal gradient algorithm. We explicitly model the update rules of the incremental proximal gradient method and develop a systematic approach to propagate the uncertainty of the solution estimate over iterations. The PIPG algorithm takes the form of Bayesian filtering updates for a state-space model constructed by using the cost function. Our framework makes it possible to utilize well-known exact or approximate Bayesian filters, such as Kalman or extended Kalman filters, to solve large-scale regularized optimization problems.

Sparse network estimation for dynamical spatio-temporal array models

Neural field models represent neuronal communication on a population level via synaptic weight functions. Using voltage sensitive dye (VSD) imaging it is possible to obtain measurements of neural fields with a relatively high spatial and temporal resolution. The synaptic weight functions represent functional connectivity in the brain and give rise to a spatio-temporal dependence structure. We present a stochastic functional differential equation for modeling neural fields, which leads to a vector autoregressive model of the data via basis expansions of the synaptic weight functions and time and space discretization. Fitting the model to data is a practical challenge as this represents a large scale regression problem. By using a 1-norm penalty in combination with localized basis functions it is possible to learn a sparse network representation of the functional connectivity of the brain, but still, the explicit construction of a design matrix can be computationally prohibitive. We demonstrate that by using tensor product basis expansions, the computation of the penalized estimator via a proximal gradient algorithm becomes feasible. It is crucial for the computations that the data is organized in an array as is the case for the three dimensional VSD imaging data. This allows for the use of array arithmetic that is both memory and time efficient. Theproposed method is implemented and showcased in the R package dynamo available from CRAN.

Trimmed Statistical Estimation via Variance Reduction

In this paper, we show how to transform any optimization problem that arises from fitting a machine learning model into one that (1) detects and removes contaminated data from the training set while (2) simultaneously fitting the trimmed model on the uncontaminated data that remains. To solve the resulting nonconvex optimization problem, we introduce a fast stochastic proximal-gradient algorithm that incorporates prior knowledge through nonsmooth regularization. For data sets of size n, our approach requires O(n2/3/ℇ) gradient evaluations to reach ℇ-accuracy, and when a certain error bound holds, the complexity improves to O(κn2/3 log(1/ℇ)), where κ is a “condition number.” These rates are n1/3 times better than those achieved by typical, nonstochastic methods.

Proximal Algorithms for Large-Scale Statistical Modeling and Sensor/Actuator Selection

Several problems in modeling and control of stochastically-driven dynamical systems can be cast as regularized semi-definite programs. We examine two such representative problems and show that they can be formulated in a similar manner. The first, in statistical modeling, seeks to reconcile observed statistics by suitably and minimally perturbing prior dynamics. The second seeks to optimally select a subset of available sensors and actuators for control purposes. To address modeling and control of large-scale systems we develop a unified algorithmic framework using proximal methods. Our customized algorithms exploit problem structure and allow handling statistical modeling, as well as sensor and actuator selection, for substantially larger scales than what is amenable to current general-purpose solvers. We establish linear convergence of the proximal gradient algorithm, draw contrast between the proposed proximal algorithms and alternating direction method of multipliers, and provide examples that illustrate the merits and effectiveness of our framework.