Convex Regularization Research Articles

Improved Low-Rank Matrix Approximation in Multivariate Case

The low-rank matrix approximation (LRMA) algorithm is an important method in signal processing. This algorithm is usually used in many tasks such as the matrix estimation and machine learning algorithms. In the past, many convex regularizers, such as the absolute-value norm, were presented for LRMA. Recently, many works show that LRMA on many nonconvex regularizers, such as the quadratic and logarithmic regularizers, can give better efficiency than the convex regularizer. Furthermore, many traditional works present LRMA in the univariate case, and do not consider in the multivariate case. Therefore, LRMA in the multivariate case on the nonconvex regularizer is presented in this work. Note that our proposed method is based on the singular value decomposition (SVD) algorithm. The relationship between singular values is considered by this multivariate case. We also present the novel nonconvex regularizer for LRMA. The simple solution for our method can be obtained from this regularizer. In many random signals, our proposed method is evaluated with the state-of-the-art algorithms. Experimental results show that the best results can be obtained from the proposed method.

Dissipative solutions to the model of a general compressible viscous fluid with the Coulomb friction law boundary condition

We study a model of a general compressible viscous fluid subject to the Coulomb friction law boundary condition. For this model, we introduce a dissipative formulation and prove the existence of dissipative solutions. The proof of this result consists of a three level approximation method: A Galerkin approximation, the classical parabolic regularization of the continuity equation as well as convex regularizations of the potential generating the viscouss stress and the boundary terms incorporating the Coulomb friction law into the dissipative formulation. This approach combines the techniques already known from the proof of the existence of dissipative solutions to a model of general compressible viscous fluids under inflow-outflow boundary conditions as well as the proof of the existence of weak solution to the incompressible Navier-Stokes equations under the Coulomb friction law boundary condition. It is the first time that this type of boundary condition is considered in the case of compressible flow.

MFILS: Tri-Selection via Convex and Nonconvex Regularizations.

In many real-world applications, data are represented by multiple instances and simultaneously associated with multiple labels. These data are always redundant and generally contaminated by different noise levels. As a result, several machine learning models fail to achieve good classification and find an optimal mapping. Feature selection, instance selection, and label selection are three effective dimensionality reduction techniques. Nevertheless, the literature was limited to feature and/or instance selection but has, to some extent, neglected label selection, which also plays an essential role in the preprocessing step, as label noises can adversely affect the performance of the underlying learning algorithms. In this article, we propose a novel framework termed multilabel Feature Instance Label Selection (mFILS) that simultaneously performs feature, instance, and label selections in both convex and nonconvex scenarios. To the best of our knowledge, this article offers, for the first time ever, a study using the triple and simultaneous selection of features, instances, and labels based on convex and nonconvex penalties in a multilabel scenario. Experimental results are built on some known benchmark datasets to validate the effectiveness of the proposed mFILS.

A Generalized Formulation for Group Selection via ADMM

This paper studies a statistical learning model where the model coefficients have a pre-determined non-overlapping group sparsity structure. We consider a combination of a loss function and a regularizer to recover the desired group sparsity patterns, which can embrace many existing works. We analyze directional stationary solutions of the proposed formulation, obtaining a sufficient condition for a directional stationary solution to achieve optimality and establishing a bound of the distance from the solution to a reference point. We develop an efficient algorithm that adopts an alternating direction method of multiplier (ADMM), showing that the iterates converge to a directional stationary solution under certain conditions. In the numerical experiment, we implement the algorithm for generalized linear models with convex and nonconvex group regularizers to evaluate the model performance on various data types, noise levels, and sparsity settings.

Regularized online exponentially concave optimization

In this paper, we investigate regularized online exponentially concave (abbr. exp-concave) optimization, in which each loss function consists of a time-varying exp-concave function and a fixed convex regularization. If the whole loss function is exp-concave, a classical method called online Newton step (ONS) enjoys an O(dlogT) regret bound, where d is the dimensionality and T is the time horizon. However, in the regularized setting, the sum of an exp-concave function and a convex regularization is not necessarily an exp-concave function, which implies that ONS is not applicable. To address this problem, we propose the proximal online Newton step (ProxONS), and show that it can attain the same O(dlogT) regret bound for any convex regularization. The main idea is to first perform an iteration of ONS with the exp-concave part in each loss function and then perform a proximal mapping with the regularization part. Furthermore, we demonstrate that by utilizing the standard online-to-batch conversion, our ProxONS can be extended to solve stochastic optimization with a regularized exp-concave objective, and enjoy an O(dlogT/T) convergence rate with high probability. Experimental results on two real datasets verify the effectiveness of our ProxONS.

Group sparsity residual constraint model with weighted log-sum penalty for image restoration

In the field of image restoration, non-convex penalties and non-local self-similarity (NSS) prior have been found to enhance image edges and textures. However, NSS-based methods often overlook the inherent variations within patch groups by treating them as a single entity for processing. Despite the existence of numerous convex or non-convex regularizers, few of them have been able to adaptively adjust penalty strength based on NSS information in order to preserve fine details in an image. To address this issue, we propose a novel group sparsity residual constraint model with weighted log-sum penalty (GSRC-Log) for image restoration in this paper. The proposed regularization term is parameterized by an adaptive scalar parameter that relates to NSS, allowing the model to prioritize high NSS group sparse coefficients while penalizing low NSS coefficients. For image denoising, we provide detailed analysis on the optimal condition of our proposed model and prove that it has a simple closed-form global solution. For image restoration tasks, we utilize the ADMM algorithm to solve the GSRC-Log problem and empirically verify its convergence. Extensive experimental results on both image denoising and compressive sensing (CS) recovery demonstrate that our proposed GSRC-Log outperforms many popular or state-of-the-art methods.

A theory of optimal convex regularization for low-dimensional recovery

Abstract We consider the problem of recovering elements of a low-dimensional model from under-determined linear measurements. To perform recovery, we consider the minimization of a convex regularizer subject to a data fit constraint. Given a model, we ask ourselves what is the ‘best’ convex regularizer to perform its recovery. To answer this question, we define an optimal regularizer as a function that maximizes a compliance measure with respect to the model. We introduce and study several notions of compliance. We give analytical expressions for compliance measures based on the best-known recovery guarantees with the restricted isometry property. These expressions permit to show the optimality of the $\ell ^{1}$-norm for sparse recovery and of the nuclear norm for low-rank matrix recovery for these compliance measures. We also investigate the construction of an optimal convex regularizer using the examples of sparsity in levels and of sparse plus low-rank models.

Convex Regularized Recursive Minimum Error Entropy Algorithm

It is well known that the recursive least squares (RLS) algorithm is renowned for its rapid convergence and excellent tracking capability. However, its performance is significantly compromised when the system is sparse or when the input signals are contaminated by impulse noise. Therefore, in this paper, the minimum error entropy (MEE) criterion is introduced into the cost function of the RLS algorithm in this paper, with the aim of counteracting the interference from impulse noise. To address the sparse characteristics of the system, we employ a universally applicable convex function to regularize the cost function. The resulting new algorithm is named the convex regularization recursive minimum error entropy (CR-RMEE) algorithm. Simulation results indicate that the performance of the CR-RMEE algorithm surpasses that of other similar algorithms, and the new algorithm excels not only in scenarios with sparse systems but also demonstrates strong robustness against pulse noise.

Convex and non-convex adaptive TV regularizations for color image restoration

Convex and non-convex adaptive TV regularizations for color image restoration

Convex Subspace Clustering by Adaptive Block Diagonal Representation.

Subspace clustering is a class of extensively studied clustering methods where the spectral-type approaches are its important subclass. Its key first step is to desire learning a representation coefficient matrix with block diagonal structure. To realize this step, many methods were successively proposed by imposing different structure priors on the coefficient matrix. These impositions can be roughly divided into two categories, i.e., indirect and direct. The former introduces the priors such as sparsity and low rankness to indirectly or implicitly learn the block diagonal structure. However, the desired block diagonalty cannot necessarily be guaranteed for noisy data. While the latter directly or explicitly imposes the block diagonal structure prior such as block diagonal representation (BDR) to ensure so-desired block diagonalty even if the data is noisy but at the expense of losing the convexity that the former's objective possesses. For compensating their respective shortcomings, in this article, we follow the direct line to propose adaptive BDR (ABDR) which explicitly pursues block diagonalty without sacrificing the convexity of the indirect one. Specifically, inspired by Convex BiClustering, ABDR coercively fuses both columns and rows of the coefficient matrix via a specially designed convex regularizer, thus naturally enjoying their merits and adaptively obtaining the number of blocks. Finally, experimental results on synthetic and real benchmarks demonstrate the superiority of ABDR to the state-of-the-arts (SOTAs).

Statistical performance of quantile tensor regression with convex regularization

In this paper, we consider high-dimensional quantile tensor regression using a general convex decomposable regularizer and analyze the statistical performances of the estimator. The rates are stated in terms of the intrinsic dimension of the estimation problem, which is, roughly speaking, the dimension of the smallest subspace that contains the true coefficient. Previously, convex regularized tensor regression has been studied with a least squares loss, Gaussian tensorial predictors and Gaussian errors, with rates that depend on the Gaussian width of a convex set. Our results extend the previous work to nonsmooth quantile loss. To deal with the non-Gaussian setting, we use the concept of Rademacher complexity with appropriate concentration inequalities instead of the Gaussian width. For the multi-linear nuclear norm penalty, our Orlicz norm bound for the operator norm of a random matrix may be of independent interest. We validate the theoretical guarantees in numerical experiments. We also demonstrate advantage of quantile regression over mean regression, and compare the performance of convex regularization method and nonconvex decomposition method in solving quantile tensor regression problem in simulation studies.

High-dimensional low-rank tensor autoregressive time series modeling

Modern technological advances have enabled an unprecedented amount of structured data with complex temporal dependence, urging the need for new methods to efficiently model and forecast high-dimensional tensor-valued time series. This paper provides a new modeling framework to accomplish this task via autoregression (AR). By considering a low-rank Tucker decomposition for the transition tensor, the proposed tensor AR can flexibly capture the underlying low-dimensional tensor dynamics, providing both substantial dimension reduction and meaningful multi-dimensional dynamic factor interpretations. For this model, we first study several nuclear-norm-regularized estimation methods and derive their non-asymptotic properties under the approximate low-rank setting. In particular, by leveraging the special balanced structure of the transition tensor, a novel convex regularization approach based on the sum of nuclear norms of square matricizations is proposed to efficiently encourage low-rankness of the coefficient tensor. To further improve the estimation efficiency under exact low-rankness, a non-convex estimator is proposed with a gradient descent algorithm, and its computational and statistical convergence guarantees are established. Simulation studies and an empirical analysis of tensor-valued time series data from multi-category import-export networks demonstrate the advantages of the proposed approach.

A Relaxation Approach to Feature Selection for Linear Mixed Effects Models

Linear Mixed-Effects (LME) models are a fundamental tool for modeling correlated data, including cohort studies, longitudinal data analysis, and meta-analysis. Design and analysis of variable selection methods for LMEs is more difficult than for linear regression because LME models are nonlinear. In this article we propose a novel optimization strategy that enables a wide range of variable selection methods for LMEs using both convex and nonconvex regularizers, including l 1 , Adaptive- l 1 , SCAD, and l 0 . The computational framework only requires the proximal operator for each regularizer to be readily computable, and the implementation is available in an open source python package pysr3, consistent with the sklearn standard. The numerical results on simulated data sets indicate that the proposed strategy improves on the state of the art for both accuracy and compute time. The variable selection techniques are also validated on a real example using a data set on bullying victimization. Supplementary materials for this article are available online.

Variable step-size convex regularized PRLS algorithms

The proportionate updating (PU) and zero-attracting (ZA) mechanisms have been applied independently in the development of sparsity-aware recursive least squares (RLS) algorithms. Recently, we propose an enhanced l1-proportionate RLS (l1-PRLS) algorithm by combining the PU and ZA mechanisms. The l1-PRLS employs a fixed step size which trades off the transient (initial convergence) and steady-state performance. In this letter, the l1-PRLS is improved in two aspects: first, we replace the l1 norm penalty by a general convex regularization (CR) function to have the CR-PRLS algorithm; second, we further introduce the variable step-size (VSS) technique to the CR-PRLS, leading to the VSS-CR-PRLS algorithm. Theoretical and numerical results were provided to corroborate the superiority of the improved algorithm.

The compressible Navier–Stokes equations with slip boundary conditions of friction type

We study a mathematical model of a viscous compressible fluid obeying the slip boundary condition of friction type. We present a notion of weak solutions to this model, in which the momentum equation and the associated energy inequality are combined into a single relation. Moreover, the slip boundary condition of friction type is incorporated into this relation by the use of a boundary integral. Our main result proves the existence of such weak solutions. The proof of this result combines the classical existence theory for the compressible Navier–Stokes equations with an approximation of the aforementioned boundary integral via a convex regularization of the absolute value function.

Recurrent Neural Network Training With Convex Loss and Regularization Functions by Extended Kalman Filtering

This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _{1}$</tex-math></inline-formula> -regularization. We show that the learning method is competitive with respect to stochastic gradient descent in a nonlinear system identification benchmark and in training a linear system with binary outputs. We also explore the use of the algorithm in data-driven nonlinear model predictive control and its relation with disturbance models for offset-free closed-loop tracking.

Training recurrent neural networks by sequential least squares and the alternating direction method of multipliers

This paper proposes a novel algorithm for training recurrent neural network models of nonlinear dynamical systems from an input/output training dataset. Arbitrary convex and twice-differentiable loss functions and regularization terms are handled by sequential least squares and either a line-search (LS) or a trust-region method of Levenberg–Marquardt (LM) type for ensuring convergence. In addition, to handle non-smooth regularization terms such as ℓ1, ℓ0, and group-Lasso regularizers, as well as to impose possibly non-convex constraints such as integer and mixed-integer constraints, we combine sequential least squares with the alternating direction method of multipliers (ADMM). We call the resulting algorithm NAILS (nonconvex ADMM iterations and least squares) in the case line search (LS) is used, or NAILM if a trust-region method (LM) is employed instead. The training method, which is also applicable to feedforward neural networks as a special case, is tested in three nonlinear system identification problems.

Interpolation of irregularly sampled seismic data via non-convex regularization

Seismic data interpolation is necessary for high-resolution imaging when the field data are not adequate or when some traces are missing. Sparse inversion is an important interpolation method for seismic data, which requires sparse transformation to build an inversion model. Most existing sparsity-promoting interpolation methods are generally based on convex function regularization, especially the L1 norm. Some studies suggest that non-convex regularization with non-convex functions can achieve better sparsity than L1 norm, and produce faster and better results. But this approach is only suitable for noise-free data interpolation, as it is less stable than convex regularization. To overcome the noisy data interpolation problem and improve the stability of non-convex regularization, we propose a novel non-convex function, called arc-tangent function, to construct the inversion model. We also present a gradient-based prediction-projection method to solve the proposed models. We used a prediction-projection method to solve the proposed models. In each iteration, we first obtained a predictive solution by gradient update algorithm and then projected it onto a convex set to update the iterative solution, and the predictive solution instead of the projection solution was output as the final results. Tests on synthetic and field data demonstrated that our non-convex regularization method performed well for both noise-free and noisy data interpolation.

On exact and robust recovery for plug-and-Play compressed sensing

Theoretical understanding of Plug-and-Play (PnP) algorithms, where an off-the-shelf denoiser is used for image regularization, is an active research topic. In this work, we study the problems of exact and stable signal recovery from compressively sensed (CS) measurements using PnP algorithms. We focus on a class of linear denoisers for which it is possible to associate a convex regularizer Φ. We consider the CS problem of minimizing Φ(x) subject to Ax=Aξ, where A is the random sensing matrix and ξ is the ground truth. We prove that if A is Gaussian and ξ lies in the range of the associated denoiser W, then the minimizer is almost surely ξ if rank(W) is less than the number of measurements and almost never otherwise. We extend the result to subgaussian matrices, except that we can guarantee exact recovery only with high probability. For noisy measurements, we consider a robust analogue of the recovery problem and prove that the error between the recovered and the ground-truth signal is bounded by the noise strength. In particular, we derive the sample complexity of CS as a function of reconstruction error and success rate. We perform numerical experiments to validate our theoretical findings.

Stochastic mirror descent method for linear ill-posed problems in Banach spaces

Consider linear ill-posed problems governed by the system for , where each A i is a bounded linear operator from a Banach space X to a Hilbert space Y i . In case p is huge, solving the problem by an iterative regularization method using the whole information at each iteration step can be very expensive, due to the huge amount of memory and excessive computational load per iteration. To solve such large-scale ill-posed systems efficiently, we develop in this paper a stochastic mirror descent method which uses only a small portion of randomly selected equations at each iteration step and incorporates convex regularization terms into the algorithm design. Therefore, our method scales very well with the problem size and has the capability of capturing features of sought solutions. The convergence property of the method depends crucially on the choice of step-sizes. We consider various rules for choosing step-sizes and obtain convergence results under a priori early stopping rules. In particular, by incorporating the spirit of the discrepancy principle we propose a choice rule of step-sizes which can efficiently suppress the oscillations in iterates and reduce the effect of semi-convergence. Furthermore, we establish an order optimal convergence rate result when the sought solution satisfies a benchmark source condition. Various numerical simulations are reported to test the performance of the method.