Convex Regularization Research Articles

Group Sparsity Residual Constraint Image Denoising Model with𝒍1/𝒍2Regularization

Group sparse residual constraint with non-local priors (GSRC) has achieved great success in image restoration producing state-of-the-art performance. In the GSRC model, the [see formula in PDF] norm minimization is employed to reduce the group sparse residual. In recent years, non-convex regularization terms have been widely used in image denoising problems, which have achieved better results in denoising than convex regularization terms. In this paper, we use the ratio of the [see formula in PDF] and [see formula in PDF] norm instead of the [see formula in PDF] norm to propose a new image denoising model, i.e., a group sparse residual constraint model with [see formula in PDF] minimization (GSRC-[see formula in PDF]). Due to the computational difficulties arisen from the non-convexity and non-linearity, we focus on a constrained optimization problem that can be solved by alternative direction method of multipliers (ADMM). Experimental results of image denoising show that the pro-posed model outperforms several state-of-the-art image denoising methods both visually and quantitatively.

Convex regularization in statistical inverse learning problems

We consider a statistical inverse learning problem, where the task is to estimate a function $ f $ based on noisy point evaluations of $ Af $, where $ A $ is a linear operator. The function $ Af $ is evaluated at i.i.d. random design points $ u_n $, $ n = 1, ..., N $ generated by an unknown general probability distribution. We consider Tikhonov regularization with general convex and $ p $-homogeneous penalty functionals and derive concentration rates of the regularized solution to the ground truth measured in the symmetric Bregman distance induced by the penalty functional. We derive concrete rates for Besov norm penalties and numerically demonstrate the correspondence with the observed rates in the context of X-ray tomography.

A Neural-Network-Based Convex Regularizer for Inverse Problems

A Neural-Network-Based Convex Regularizer for Inverse Problems

Learned Reconstruction Methods With Convergence Guarantees: A survey of concepts and applications

In recent years, deep learning has achieved remarkable empirical success for image reconstruction. This has catalyzed an ongoing quest for the precise characterization of the correctness and reliability of data-driven methods in critical use cases, for instance, in medical imaging. Notwithstanding the excellent performance and efficacy of deep learning-based methods, concerns have been raised regarding the approaches’ stability, or lack thereof, with serious practical implications. Significant advances have been made in recent years to unravel the inner workings of data-driven image recovery methods, challenging their widely perceived black-box nature. In this article, we specify relevant notions of convergence for data-driven image reconstruction, which forms the basis of a survey of learned methods with mathematically rigorous reconstruction guarantees. An example that is highlighted is the role of input-convex neural networks (ICNNs), offering the possibility to combine the power of deep learning with classical convex regularization theory for devising methods that are provably convergent. This survey article is aimed at both methodological researchers seeking to advance the frontiers of our understanding of data-driven image reconstruction methods as well as practitioners by providing an accessible description of useful convergence concepts and by placing some of the existing empirical practices on a solid mathematical foundation.

Sparse Regularized Optimal Transport with Deformed q-Entropy.

Optimal transport is a mathematical tool that has been a widely used to measure the distance between two probability distributions. To mitigate the cubic computational complexity of the vanilla formulation of the optimal transport problem, regularized optimal transport has received attention in recent years, which is a convex program to minimize the linear transport cost with an added convex regularizer. Sinkhorn optimal transport is the most prominent one regularized with negative Shannon entropy, leading to densely supported solutions, which are often undesirable in light of the interpretability of transport plans. In this paper, we report that a deformed entropy designed by q-algebra, a popular generalization of the standard algebra studied in Tsallis statistical mechanics, makes optimal transport solutions supported sparsely. This entropy with a deformation parameter q interpolates the negative Shannon entropy (q=1) and the squared 2-norm (q=0), and the solution becomes more sparse as q tends to zero. Our theoretical analysis reveals that a larger q leads to a faster convergence when optimized with the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. In summary, the deformation induces a trade-off between the sparsity and convergence speed.

A nonconvex hybrid regularization model for restoring blurred images with mixed noises

Nonconvex regularization has advantages over convex regularization for preserving the edges of images, and hybrid regularization method based on total variation and wavelet frames is also highly effective for improving the quality of restoring images. Therefore, this paper proposes a nonconvex hybrid regularization model for restoring blurred images with additive noises and multiplicative noises, which is a quite challenging problem compared with the problem of restoring blurred images with a single type of noise. Although the proposed model is jointly nonconvex, every subproblem is convex under some mild conditions when the alternating minimization method is used to solve the proposed model. At the same time, the convergence property of the corresponding algorithm is discussed. Numerical experiments show that the proposed model is effective in edge-preserving, staircase-reduction and detail-reserving in image restoration and it outperforms the state-of-the-art model in terms of the PSNR, ReErr and MSSIM values.

Adaptive Rank and Structured Sparsity Corrections for Hyperspectral Image Restoration

Hyperspectral images (HSIs) are inevitably contaminated by the mixed noise (such as Gaussian noise, impulse noise, deadlines, and stripes), which could influence the subsequent processing accuracy. Generally, HSI restoration can be transformed into the low-rank matrix recovery (LRMR). In the LRMR, the nuclear norm is widely used to substitute the matrix rank, but its effectiveness is still worth improving. Besides, the l0 -norm cannot capture the sparse noise's structured sparsity property. To handle these issues, the adaptive rank and structured sparsity corrections (ARSSC) are presented for HSI restoration. The ARSSC introduces two convex regularizers, that is: 1) the rank correction (RC) and 2) the structured sparsity correction (SSC), to, respectively, approximate the matrix rank and the l2,0 -norm. The RC and the SSC can adaptively offset the penalization of large entries from the nuclear norm and the l2,1 -norm, respectively, where the larger the entry, the greater its offset. Therefore, the proposed ARSSC achieves a tighter approximation of the noise-free HSI low-rank structure and promotes the structured sparsity of sparse noise. An efficient alternative direction method of multipliers (ADMM) algorithm is applied to solve the resulting convex optimization problem. The superiority of the ARSSC in terms of the mixed noise removal and spatial-spectral structure information preserving, is demonstrated by several experimental results both on simulated and real datasets, compared with other state-of-the-art HSI restoration approaches.

A Second Order Primal-Dual Method for Nonsmooth Convex Composite Optimization

We develop a second order primal-dual method for optimization problems in which the objective function is given by the sum of a strongly convex twice differentiable term and a possibly nondifferentiable convex regularizer. After introducing an auxiliary variable, we utilize the proximal operator of the nonsmooth regularizer to transform the associated augmented Lagrangian into a function that is once, but not twice, continuously differentiable. The saddle point of this function corresponds to the solution of the original optimization problem. We employ a generalization of the Hessian to define second-order updates on this function and prove global exponential stability of the corresponding differential inclusion. Furthermore, we develop a globally convergent customized algorithm that utilizes the primal-dual augmented Lagrangian as a merit function. We show that the search direction can be computed efficiently and prove quadratic/superlinear asymptotic convergence. We use the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _1$</tex-math></inline-formula> -regularized model predictive control problem and the problem of designing a distributed controller for a spatially invariant system to demonstrate the merits and the effectiveness of our method.

Double bias correction for high-dimensional sparse additive hazards regression with covariate measurement errors.

We propose an inferential procedure for additive hazards regression with high-dimensional survival data, where the covariates are prone to measurement errors. We develop a double bias correction method by first correcting the bias arising from measurement errors in covariates through an estimating function for the regression parameter. By adopting the convex relaxation technique, a regularized estimator for the regression parameter is obtained by elaborately designing a feasible loss based on the estimating function, which is solved via linear programming. Using the Neyman orthogonality, we propose an asymptotically unbiased estimator which further corrects the bias caused by the convex relaxation and regularization. We derive the convergence rate of the proposed estimator and establish the asymptotic normality for the low-dimensional parameter estimator and the linear combination thereof, accompanied with a consistent estimator for the variance. Numerical experiments are carried out on both simulated and real datasets to demonstrate the promising performance of the proposed double bias correction method.

Screening for a Reweighted Penalized Conditional Gradient Method

The conditional gradient method (CGM) is widely used in large-scale sparse convex optimization, having a low per iteration computational cost for structured sparse regularizers and a greedy approach for collecting nonzeros. We explore the sparsity acquiring properties of a general penalized CGM (P-CGM) for convex regularizers and a reweighted penalized CGM (RP-CGM) for nonconvex regularizers, replacing the usual convex constraints with gauge-inspired penalties. This generalization does not increase the per-iteration complexity noticeably. Without assuming bounded iterates or using line search, we show O(1/t) convergence of the gap of each subproblem, which measures distance to a stationary point. We couple this with a screening rule which is safe in the convex case, converging to the true support at a rate O(1/(δ 2 )) where δ≥0 measures how close the problem is to degeneracy. In the nonconvex case the screening rule converges to the true support in a finite number of iterations, but is not necessarily safe in the intermediate iterates. In our experiments, we verify the consistency of the method and adjust the aggressiveness of the screening rule by tuning the concavity of the regularizer.

Distributed Stochastic Inertial-Accelerated Methods with Delayed Derivatives for Nonconvex Problems

Stochastic gradient methods (SGMs) are predominant approaches for solving stochastic optimization. On smooth nonconvex problems, a few acceleration techniques have been applied to improve the convergence rate of SGMs. However, little exploration has been made on applying a certain acceleration technique to a stochastic subgradient method (SsGM) for nonsmooth nonconvex problems. In addition, few efforts have been made to analyze an (accelerated) SsGM with delayed derivatives. The information delay naturally happens in a distributed system, where computing workers do not coordinate with each other. In this paper, we propose an inertial proximal SsGM for solving nonsmooth nonconvex stochastic optimization problems. The proposed method can have guaranteed convergence even with delayed derivative information in a distributed environment. Convergence rate results are established for three classes of nonconvex problems: weakly convex nonsmooth problems with a convex regularizer, composite nonconvex problems with a nonsmooth convex regularizer, and smooth nonconvex problems. For each problem class, the convergence rate is $O(1/K^{\frac{1}{2}})$ in the expected value of the gradient norm square, for $K$ iterations. In a distributed environment, the convergence rate of the proposed method will be slowed down by the information delay. Nevertheless, the slow-down effect will decay with the number of iterations for the latter two problem classes. We test the proposed method on three applications. The numerical results clearly demonstrate the advantages of using the inertial-based acceleration. Furthermore, we observe higher parallelization speed-up in asynchronous updates over the synchronous counterpart, though the former uses delayed derivatives. Our source code is available at https://github.com/RPI-OPT/Inertial-SsGM.

Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes

We present new policy mirror descent (PMD) methods for solving reinforcement learning (RL) problems with either strongly convex or general convex regularizers. By exploring the structural properties of these overall highly nonconvex problems we show that the PMD methods exhibit fast linear rate of convergence to the global optimality. We develop stochastic counterparts of these methods, and establish an $${{\mathcal {O}}}(1/\epsilon )$$ (resp., $${{\mathcal {O}}}(1/\epsilon ^2)$$ ) sampling complexity for solving these RL problems with strongly (resp., general) convex regularizers using different sampling schemes, where $$\epsilon $$ denote the target accuracy. We further show that the complexity for computing the gradients of these regularizers, if necessary, can be bounded by $${{\mathcal {O}}}\{(\log _\gamma \epsilon ) [(1-\gamma )L/\mu ]^{1/2}\log (1/\epsilon )\}$$ (resp., $${{\mathcal {O}}} \{(\log _\gamma \epsilon ) (L/\epsilon )^{1/2}\}$$ ) for problems with strongly (resp., general) convex regularizers. Here $$\gamma $$ denotes the discounting factor. To the best of our knowledge, these complexity bounds, along with our algorithmic developments, appear to be new in both optimization and RL literature. The introduction of these convex regularizers also greatly enhances the flexibility and thus expands the applicability of RL models.

Fundamental barriers to high-dimensional regression with convex penalties

In high-dimensional regression, we attempt to estimate a parameter vector β0∈Rp from n≲p observations {(yi,xi)}i≤n, where xi∈Rp is a vector of predictors and yi is a response variable. A well-established approach uses convex regularizers to promote specific structures (e.g., sparsity) of the estimate βˆ while allowing for practical algorithms. Theoretical analysis implies that convex penalization schemes have nearly optimal estimation properties in certain settings. However, in general the gaps between statistically optimal estimation (with unbounded computational resources) and convex methods are poorly understood. We show that when the statistican has very simple structural information about the distribution of the entries of β0, a large gap frequently exists between the best performance achieved by any convex regularizer satisfying a mild technical condition and either: (i) the optimal statistical error or (ii) the statistical error achieved by optimal approximate message passing algorithms. Remarkably, a gap occurs at high enough signal-to-noise ratio if and only if the distribution of the coordinates of β0 is not log-concave. These conclusions follow from an analysis of standard Gaussian designs. Our lower bounds are expected to be generally tight, and we prove tightness under certain conditions.

Structure-Guaranteed SAR Imagery via Spatially-Variant Morphology Regularization in ADMM Manner

Conventional sparsity-driven synthetic aperture radar (SAR) imagery proceeds via &#x2113;₁ regularization, or named by least absolute shrinkage and selection operator (LASSO). However, followed by the enhanced sparse feature, structures of the scenes or targets of interests in weak scattering are easily lost. Therefore, it becomes difficult to make use of the high-resolution SAR data, even high costs have been paid for the resolution. In this paper, a novel structure-guaranteed SAR (SG-SAR) imaging algorithm is proposed by utilizing the morphology metric for the cluster feature of the scatterers of the scenes/targets of interests. By introducing structural prior in terms of morphology norm, the intended structure features can be highlighted via convex regularization. More specifically, to accommodate to complicated scenarios or targets, the structure element in the morphology regularizer is designed to be spatially variant under structure tensor representation. Different from conventional convex optimizations, the proposed SG-SAR algorithm is solved under alternating direction method of multipliers (ADMM), which is flexible to incorporate with the super-resolution imagery. In such cases, both sparse and structural features can be simultaneously enhanced, even with limited measurements and in low signal-to-noise ratio (SNR). Superior convergence and robustness can be guaranteed. Moreover, a grouping mask scheme is used to accommodate to the complex-valued SAR data. Finally, both simulated and measured SAR data are applied for the validation. Comparisons with the conventions are performed in terms of phase transition analysis, so as to verify the superiority of the proposed algorithm both qualitatively and quantitatively.

Distributed Sparse Optimization With Weakly Convex Regularizer: Consensus Promoting and Approximate Moreau Enhanced Penalties Towards Global Optimality

We propose a distributed optimization framework to generate accurate sparse estimates while allowing an algorithmic solution with guaranteed convergence to a global minimizer. To this end, the proposed problem formulation involves the minimax concave penalty together with an additional penalty called consensus promoting penalty (CPP) that induces convexity to the resulting optimization problem. This problem is solved with an exact first-order proximal gradient algorithm, which employs a pair of proximity operators and is referred to as the distributed proximal and debiasing-gradient (DPD) method. Numerical examples show that CPP not only convexifies the whole cost function, but it also accelerates the convergence speed with respect to the system mismatch.

Guided Nonlocal Patch Regularization and Efficient Filtering-Based Inversion for Multiband Fusion

In multiband fusion, an image with a high spatial and low spectral resolution is combined with an image with a low spatial but high spectral resolution to produce a single multiband image having high spatial and spectral resolutions. This comes up in remote sensing applications such as pansharpening (MS+PAN), hyperspectral sharpening (HS+PAN), and HS-MS fusion (HS+MS). Remote sensing images are textured and have repetitive structures. Motivated by nonlocal patch-based methods for image restoration, we propose a convex regularizer that (i) takes into account long-distance correlations, (ii) penalizes patch variation, which is more effective than pixel variation for capturing texture information, and (iii) uses the higher spatial resolution image as a guide image for weight computation. We come up with an efficient ADMM algorithm for optimizing the regularizer along with a standard least-squares loss function derived from the imaging model. The novelty of our algorithm is that by expressing patch variation as filtering operations and by judiciously splitting the original variables and introducing latent variables, we are able to solve the ADMM subproblems efficiently using FFT-based convolution and soft-thresholding. As far as the reconstruction quality is concerned, our method is shown to outperform state-of-the-art variational and deep learning techniques.

ADMM-ADAM: A New Inverse Imaging Framework Blending the Advantages of Convex Optimization and Deep Learning

Alternating direction method of multipliers (ADMM) and adaptive moment estimation (ADAM) are two optimizers of paramount importance in convex optimization (CO) and deep learning (DL), respectively. Numerous state-of-the-art algorithms for solving inverse problems are achieved by carefully designing a convex criterion, typically composed of a data-fitting term and a regularizer. Even when the regularizer is convex, its mathematical form is often sophisticated, hence inducing a math-heavy optimization procedure and making the algorithm design a daunting task for software engineers. Probably for this reason, people turn to solve the inverse problems via DL, but this requires big data collection, quite time-consuming if not impossible. Motivated by these facts, we propose a new framework, termed as ADMM-ADAM, for solving inverse problems. As the key contribution, even just with small/single data, the proposed ADMM-ADAM is able to exploit DL to obtain a convex regularizer of very simple math form, followed by solving the regularized criterion using simple CO algorithm. As a side contribution, a state-of-the-art hyperspectral inpainting algorithm is designed under ADMM-ADAM, demonstrating its superiority even without the aid of big data or sophisticated mathematical regularization.

Penalized wavelet estimation and robust denoising for irregular spaced data

Nonparametric univariate regression via wavelets is usually implemented under the assumptions of dyadic sample size, equally spaced fixed sample points, and i.i.d. normal errors. In this work, we propose, study and compare some wavelet based nonparametric estimation methods designed to recover a one-dimensional regression function for data that not necessary possess the above requirements. These methods use appropriate regularizations by penalizing the decomposition of the unknown regression function on a wavelet basis of functions evaluated on the sampling design. Exploiting the sparsity of wavelet decompositions for signals belonging to homogeneous Besov spaces, we use some efficient proximal gradient descent algorithms, available in recent literature, for computing the estimates with fast computation times. Our wavelet based procedures, in both the standard and the robust regression case have favorable theoretical properties, thanks in large part to the separability nature of the (non convex) regularization they are based on. We establish asymptotic global optimal rates of convergence under weak conditions. It is known that such rates are, in general, unattainable by smoothing splines or other linear nonparametric smoothers. Lastly, we present several experiments to examine the empirical performance of our procedures and their comparisons with other proposals available in the literature. An interesting regression analysis of some real data applications using these procedures unambiguously demonstrate their effectiveness.

Levenberg–Marquardt method for ill-posed inverse problems with possibly non-smooth forward mappings between Banach spaces

In this paper, we consider a Levenberg–Marquardt method with general regularization terms that are uniformly convex on bounded sets to solve the ill-posed inverse problems in Banach spaces, where the forward mapping might not Gâteaux differentiable and the image space is unnecessarily reflexive. The method therefore extends the one proposed by Jin and Yang in (2016 Numer. Math. 133 655–684) for smooth inverse problem setting with globally uniformly convex regularization terms. We prove a novel convergence analysis of the proposed method under some standing assumptions, in particular, the generalized tangential cone condition and a compactness assumption. All these assumptions are fulfilled when investigating the identification of the heat source for semilinear elliptic boundary-value problems with a Robin boundary condition, a heat source acting on the boundary, and a possibly non-smooth nonlinearity. Therein, the Clarke subdifferential of the non-smooth nonlinearity is employed to construct the family of bounded operators that is a replacement for the non-existing Gâteaux derivative of the forward mapping. The efficiency of the proposed method is illustrated with a numerical example.

A First-Order Optimization Algorithm for Statistical Learning with Hierarchical Sparsity Structure

In many statistical learning problems, it is desired that the optimal solution conform to an a priori known sparsity structure represented by a directed acyclic graph. Inducing such structures by means of convex regularizers requires nonsmooth penalty functions that exploit group overlapping. Our study focuses on evaluating the proximal operator of the latent overlapping group lasso developed by Jacob et al. in 2009 . We implemented an alternating direction method of multiplier with a sharing scheme to solve large-scale instances of the underlying optimization problem efficiently. In the absence of strong convexity, global linear convergence of the algorithm is established using the error bound theory. More specifically, the paper contributes to establishing primal and dual error bounds when the nonsmooth component in the objective function does not have a polyhedral epigraph. We also investigate the effect of the graph structure on the speed of convergence of the algorithm. Detailed numerical simulation studies over different graph structures supporting the proposed algorithm and two applications in learning are provided. Summary of Contribution: The paper proposes a computationally efficient optimization algorithm to evaluate the proximal operator of a nonsmooth hierarchical sparsity-inducing regularizer and establishes its convergence properties. The computationally intensive subproblem of the proposed algorithm can be fully parallelized, which allows solving large-scale instances of the underlying problem. Comprehensive numerical simulation studies benchmarking the proposed algorithm against five other methods on the speed of convergence to optimality are provided. Furthermore, performance of the algorithm is demonstrated on two statistical learning applications related to topic modeling and breast cancer classification. The code along with the simulation studies and benchmarks are available on the corresponding author’s GitHub website for evaluation and future use.