Penalized Least-squares Problem Research Articles

On the Convergence of the SINDy Algorithm

One way to understand time-series data is to identify the underlying dynamical system which generates it. This task can be done by selecting an appropriate model and a set of parameters which best fits the dynamics while providing the simplest representation (i.e. the smallest amount of terms). One such approach is the sparse identification of nonlinear dynamics framework [6] which uses a sparsity-promoting algorithm that iterates between a partial least-squares fit and a thresholding (sparsity-promoting) step. In this work, we provide some theoretical results on the behavior and convergence of the algorithm proposed in [6]. In particular, we prove that the algorithm approximates local minimizers of an unconstrained $\ell^0$-penalized least-squares problem. From this, we provide sufficient conditions for general convergence, rate of convergence, and conditions for one-step recovery. Examples illustrate that the rates of convergence are sharp. In addition, our results extend to other algorithms related to the algorithm in [6], and provide theoretical verification to several observed phenomena.

Some Results on ℓ1 Polynomial Trend Filtering

ℓ1 polynomial trend filtering, which is a filtering method described as an ℓ1-norm penalized least-squares problem, is promising because it enables the estimation of a piecewise polynomial trend in a univariate economic time series without prespecifying the number and location of knots. This paper shows some theoretical results on the filtering, one of which is that a small modification of the filtering provides not only identical trend estimates as the filtering but also extrapolations of the trend beyond both sample limits.

On the choice of subspace for large-scale Tikhonov regularization problems in general form

Many applications in science and engineering require the solution of large linear discrete ill-posed problems. The matrices that define these problems are very ill-conditioned and possibly numerically singular, and the right-hand sides, which represent the measured data, typically are contaminated by measurement error. Straightforward solution of these problems generally is not meaningful due to severe error propagation. Tikhonov regularization seeks to alleviate this difficulty by replacing the given linear discrete ill-posed problem by a penalized least-squares problem, whose solution is less sensitive to the error in the right-hand side. The penalty term is determined by a regularization matrix. A suitable choice of this matrix may result in a computed solution of higher quality than when the regularization matrix is the identity. Two iterative solution methods based on the global Arnoldi decomposition method have been proposed in the literature for the solution of large-scale penalized least-squares problems that stem from Tikhonov regularization. In one of these, the regularization matrix influences the choice of the solution subspace; in the other one, it does not. This paper compares these approaches both with respect to the quality of the computed solution and computing time.

Fractional Tikhonov regularization with a nonlinear penalty term

Tikhonov regularization is one of the most popular methods for solving linear systems of equations or linear least-squares problems with a severely ill-conditioned matrix and an error-contaminated data vector (right-hand side). This regularization method replaces the given problem by a penalized least-squares problem. It is well known that Tikhonov regularization in standard form may yield approximate solutions that are too smooth, i.e., the computed approximate solution may lack many details that the desired solution of the associated, but unavailable, error-free problem might possess. Fractional Tikhonov regularization methods have been introduced to remedy this shortcoming. However, the computed solution determined by fractional Tikhonov methods in standard form may display undesirable spurious oscillations. This paper proposes that fractional Tikhonov methods be equipped with a nonlinear penalty term, such as a TV-norm penalty term, to reduce unwanted oscillations. Numerical examples illustrate the benefits of this approach.

On the reduction of Tikhonov minimization problems and the construction of regularization matrices

Tikhonov regularization replaces a linear discrete ill-posed problem by a penalized least-squares problem, whose solution is less sensitive to errors in the data and round-off errors introduced during the solution process. The penalty term is defined by a regularization matrix and a regularization parameter. The latter generally has to be determined during the solution process. This requires repeated solution of the penalized least-squares problem. It is therefore attractive to transform the least-squares problem to simpler form before solution. The present paper describes a transformation of the penalized least-squares problem to simpler form that is faster to compute than available transformations in the situation when the regularization matrix has linearly dependent columns and no exploitable structure. Properties of this kind of regularization matrices are discussed and their performance is illustrated.

Fractional Tikhonov regularization for linear discrete ill-posed problems

Tikhonov regularization is one of the most popular methods for solving linear systems of equations or linear least-squares problems with a severely ill-conditioned matrix A. This method replaces the given problem by a penalized least-squares problem. The present paper discusses measuring the residual error (discrepancy) in Tikhonov regularization with a seminorm that uses a fractional power of the Moore-Penrose pseudoinverse of AA T as weighting matrix. Properties of this regularization method are discussed. Numerical examples illustrate that the proposed scheme for a suitable fractional power may give approximate solutions of higher quality than standard Tikhonov regularization.

Recovering Sparse Signals With a Certain Family of Nonconvex Penalties and DC Programming

This paper considers the problem of recovering a sparse signal representation according to a signal dictionary. This problem could be formalized as a penalized least-squares problem in which sparsity is usually induced by a lscr1-norm penalty on the coefficients. Such an approach known as the Lasso or Basis Pursuit Denoising has been shown to perform reasonably well in some situations. However, it was also proved that nonconvex penalties like the pseudo lscrq-norm with q < 1 or smoothly clipped absolute deviation (SCAD) penalty are able to recover sparsity in a more efficient way than the Lasso. Several algorithms have been proposed for solving the resulting nonconvex least-squares problem. This paper proposes a generic algorithm to address such a sparsity recovery problem for some class of nonconvex penalties. Our main contribution is that the proposed methodology is based on an iterative algorithm which solves at each iteration a convex weighted Lasso problem. It relies on the family of nonconvex penalties which can be decomposed as a difference of convex functions (DC). This allows us to apply DC programming which is a generic and principled way for solving nonsmooth and nonconvex optimization problem. We also show that several algorithms in the literature dealing with nonconvex penalties are particular instances of our algorithm. Experimental results demonstrate the effectiveness of the proposed generic framework compared to existing algorithms, including iterative reweighted least-squares methods.

Detection of Subsurface Bubbles with Discrete Electromagnetic Geotomography

Abstract In electromagnetic geotomography, the inversion is very ill-posed due to the limita- tion of an angular range of projections, and consequently the reconstructed image is different than the true image. Nevertheless, the reconstruction can be much im- proved if we are interested only in a piece of information on the interior volume of the examined object, e.g. detection of some anomalies (bubbles). We assume that the image can be represented by binary values, hence: Discrete Electromagnetic Geotomography (DEG). In this paper, we discuss two discrete image reconstruction algorithms. The first one uses the D.C. programming to find the integer solution to the penalized least-squares problem. The penalty term is based on the Gibbs distribution. In the second one, the mean field annealing is applied to maximize the Gibbs-Boltzmann distribution associated with the penalized least-squares function. The numerical results are presented for noise-free and noisy data.