Solution Of Linear Inverse Problem Research Articles

The Sparsity of Cycle Spinning for Wavelet-Based Solutions of Linear Inverse Problems

The usual explanation of the efficacy of wavelet-based methods hinges on the sparsity of many real-world objects in the wavelet domain. Yet, standard wavelet-shrinkage techniques for sparse reconstruction are not competitive in practice, one reason being that the lack of shift-invariance of the wavelet transform produces blocky artifacts. The standard remedy is cycle spinning, which results in a substantial reduction of these artifacts. In this letter, we propose a theoretical investigation of the sparsity of solutions to the cycle-spinning variant of wavelet-based resolutions of linear inverse problems. We derive a representer theorem that provides a complete characterization of the solution set. Our theorem indicates that the solutions are typically not sparse, where sparsity is measured with respect to the wavelet dictionary. This exposes that the role of sparsity in the success of wavelet-based solutions of linear inverse problems requires further investigation. We corroborate our theoretical results with numerical examples for the problem of image denoising.

On the Strong Convergence of Forward-Backward Splitting in Reconstructing Jointly Sparse Signals

We consider the problem of reconstructing an infinite set of sparse, finite-dimensional vectors, that share a common sparsity pattern, from incomplete measurements. This is in contrast to the work (Daubechies et al., Pure Appl. Math. 57(11), 1413–1457, 2004), where the single vector signal can be infinite-dimensional, and (Fornasier and Rauhut, SIAM J. Numer. Anal. 46(2), 577613, 2008), which extends the aforementioned work to the joint sparse recovery of finite number of infinite-dimensional vectors. In our case, to take account of the joint sparsity and promote the coupling of nonvanishing components, we employ a convex relaxation approach with mixed norm penalty ℓ2,1. This paper discusses the computation of the solutions of linear inverse problems with such relaxation by a forward-backward splitting algorithm. However, since the solution matrix possesses infinitely many columns, the arguments of Daubechies et al. (Pure Appl. Math. 57(11), 1413–1457, 2004) no longer apply. As such, we establish new strong convergence results for the algorithm, in particular when the set of jointly sparse vectors is infinite.

A model reduction approach for inverse problems with operator valued data

We study the efficient numerical solution of linear inverse problems with operator valued data which arise, e.g., in seismic exploration, inverse scattering, or tomographic imaging. The high-dimensionality of the data space implies extremely high computational cost already for the evaluation of the forward operator which makes a numerical solution of the inverse problem, e.g., by iterative regularization methods, practically infeasible. To overcome this obstacle, we take advantage of the underlying tensor product structure of the problem and propose a strategy for constructing low-dimensional certified reduced order models of quasi-optimal rank for the forward operator which can be computed much more efficiently than the truncated singular value decomposition. A complete analysis of the proposed model reduction approach is given in a functional analytic setting and the efficient numerical construction of the reduced order models as well as of their application for the numerical solution of the inverse problem is discussed. In summary, the setup of a low-rank approximation can be achieved in an offline stage at essentially the same cost as a single evaluation of the forward operator, while the actual solution of the inverse problem in the online phase can be done with extremely high efficiency. The theoretical results are illustrated by application to a typical model problem in fluorescence optical tomography.

Efficient solution of linear inverse problems using an iterative linear neural network with a generalization training approach

Inverse problems are some of the most important mathematical problems in science and mathematics because their solution yields information about parameters that are not directly observable. Artificial neural networks have long been used as a mathematical modelling method and have been used successfully to solve inverse problems for application including denoising and medical image reconstruction. Many inverse problems result from integral processes that can be modelled using a linear formulation. These can be efficiently solved via simple networks which are easily trained with reasonable datasets. An innovative simple neural network architecture, the iterative linear neural network (ILNN), consisting of two non-hidden layer networks, one for the forward model and one for the inverse model, is proposed to solve linear inverse problems. Iteration between the two models refines network outcomes with greater accuracy than a network with only the inverse model. A training procedure accompanying the network is also introduced. The network needs to train only the inverse model with one-hot vectors as targets. The training inputs of the inverse model define the weights of the forward model. The number of targets is finite and equal to the length of the vector. With the defined targets, the training process ensures that the inverse model is at least a left inverse of the forward model. This leads to generalizable networks. The experimental results show that the ILNN produces good results even if its inverse model is not perfectly trained. The proposed network is applied to solve two linear inverse problems, deconvolution and the inverse Radon transform. The network successfully reconstructed original data following blurring and Radon transformation.

Accelerated Iterative Regularization via Dual Diagonal Descent

We propose and analyze an accelerated iterative dual diagonal descent algorithm for the solution of linear inverse problems with general regularization and data-fit functions. In particular , we develop an inertial approach of which we analyze both convergence and stability. Using tools from inexact proximal calculus, we prove early stopping results with optimal convergence rates for additive data-fit terms as well as more general cases, such as the Kullback-Leibler divergence, for which different type of proximal point approximations hold.

Linear convergence of inexact descent method and inexact proximal gradient algorithms for lower-order regularization problems

The $$\ell _p$$ regularization problem with $$0< p< 1$$ has been widely studied for finding sparse solutions of linear inverse problems and gained successful applications in various mathematics and applied science fields. The proximal gradient algorithm is one of the most popular algorithms for solving the $$\ell _p$$ regularisation problem. In the present paper, we investigate the linear convergence issue of one inexact descent method and two inexact proximal gradient algorithms (PGA). For this purpose, an optimality condition theorem is explored to provide the equivalences among a local minimum, second-order optimality condition and second-order growth property of the $$\ell _p$$ regularization problem. By virtue of the second-order optimality condition and second-order growth property, we establish the linear convergence properties of the inexact descent method and inexact PGAs under some simple assumptions. Both linear convergence to a local minimal value and linear convergence to a local minimum are provided. Finally, the linear convergence results of these methods are extended to the infinite-dimensional Hilbert spaces. Our results cannot be established under the framework of Kurdyka–Łojasiewicz theory.

Iterative positive thresholding algorithm for non-negative sparse optimization

The non-negative regularization problem has been widely studied for finding non-negative sparse solutions of linear inverse problems and gained successful applications in various application areas. In the present paper, we propose an iterative positive thresholding algorithm (IPTA) to solve the non-negative regularization problem and investigate its convergence properties in finite- or infinite-dimensional Hilbert spaces. The significant advantage of the IPTA is that it is very simple and of low computation cost, and thus, it is practically attractive, especially for large-scale problems. The global convergence of the IPTA is achieved under some mild assumptions on algorithmic parameters. Furthermore, we introduce a notion of positive orthogonal sparsity pattern, and use it to establish the linear convergence rate of the IPTA to a global minimum. Finally, the numerical study on compressive sensing shows that the proposed IPTA is efficient in approaching the non-negative sparse solutions of linear inverse problems and outperforms several existing algorithms in sparse optimization.

Continuous-Domain Solutions of Linear Inverse Problems With Tikhonov Versus Generalized TV Regularization

We consider linear inverse problems that are formulated in the continuous domain. The object of recovery is a function that is assumed to minimize a convex objective functional. The solutions are constrained by imposing a continuous-domain regularization. We derive the parametric form of the solution (representer theorems) for Tikhonov (quadratic) and generalized total-variation (gTV) regularizations. We show that, in both cases, the solutions are splines that are intimately related to the regularization operator. In the Tikhonov case, the solution is smooth and constrained to live in a fixed subspace that depends on the measurement operator. By contrast, the gTV regularization results in a sparse solution composed of only a few dictionary elements that are upper-bounded by the number of measurements and independent of the measurement operator. Our findings for the gTV regularization resonates with the minimization of the $l_1$ norm, which is its discrete counterpart and also produces sparse solutions. Finally, we find the experimental solutions for some measurement models in one dimension. We discuss the special case when the gTV regularization results in multiple solutions and devise an algorithm to find an extreme point of the solution set which is guaranteed to be sparse.

Bayes Meets Krylov: Statistically Inspired Preconditioners for CGLS

The solution of linear inverse problems when the unknown parameters outnumber data requires addressing the problem of a nontrivial null space. After restating the problem within the Bayesian framework, a priori information about the unknown can be utilized for determining the null space contribution to the solution. More specifically, if the solution of the associated linear system is computed by the conjugate gradient for least squares (CGLS) method, the additional information can be encoded in the form of a right preconditioner. In this paper we study how the right preconditioner changes the Krylov subspaces where the CGLS iterates live, and we draw a tighter connection between Bayesian inference and Krylov subspace methods. The advantages of a Bayes-meets-Krylov approach to the solution of underdetermined linear inverse problems is illustrated with computed examples.

Splines Are Universal Solutions of Linear Inverse Problems with Generalized TV Regularization

Splines come in a variety of flavors that can be characterized in terms of some differential operator L. The simplest piecewise-constant model corresponds to the derivative operator. Likewise, one can extend the traditional notion of total variation by considering more general operators than the derivative. This leads us to the definition of the generalized Beppo-Levi space M, which is further identified as the direct sum of two Banach spaces. We then prove that the minimization of the generalized total variation (gTV) over M, subject to some arbitrary (convex) consistency constraints on the linear measurements of the signal, admits nonuniform L-spline solutions with fewer knots than the number of measurements. This shows that non-uniform splines are universal solutions of continuous-domain linear inverse problems with LASSO, L1, or TV-like regularization constraints. Remarkably, the spline-type is fully determined by the choice of L and does not depend on the actual nature of the measurements.

Research Spotlights

This issue's Research Spotlights section contains two papers. The first of these is “Splines Are Universal Solutions of Linear Inverse Problems with Generalized TV Regularization," by Michael Unser, Julien Fageot, and John Paul Ward. The paper shows that for certain minimization problems in the $L_1$ norm or total variation seminorm over a continuous domain, there exist spline solutions. Further, these solutions have fewer points of discontinuity (knots) than there are measurements. Careful attention is given to the presentation of a particularly useful case of this theorem, namely, that of sampling and compressed sensing, as used in the reconstruction of biomedical images, or more generally in the recovery of a signal from a sparse set of measurements. This work is a step along the way towards a universality result for spline solutions, and thus has the potential to change methods of analysis and discretization of such problems. It will be of interest to those interested both in the theory for minimization problems and those interested in applications of such problems in applied mathematics and beyond. The second paper in this section, “The Nonnegative Rank of a Matrix: Hard Problems, Easy Solutions," by Yaroslav Shitov, presents an elementary linear algebra technique based on the nonnegative rank of a matrix. The nonnegative rank of a matrix $A$ is the minimum number of nonnegative rank-one matrices required to sum to $A$. This paper demonstrates that this rank is $NP$-hard to compute, as was previously shown using other techniques in 1993 by Jiang and Ravikumar and in 2009 by Vavasis. In 1993, Cohen and Rothblum asked whether it is possible to write a nonnegative rank $k$ matrix as a sum of $k$ rational nonnegative rank-one matrices. The problem had remained open since that time. This paper demonstrates that the answer to this question is, in general, no: it contains an example of a $21 \times 21$ nonnegative rank-one matrix which cannot be decomposed using rational nonnegative rank-one matrices. Note that this paper appeared on the arXiv simultaneously with another paper which also solved the Cohen--Rothblum problem. Although the paper is only seven pages long, it manages to give a detailed account of the methods used. It will be of primary interest to those in numerical linear algebra. However, since it uses elementary techniques, it is accessible to readers outside the field.

Penalty Functions Derived From Monotone Mappings

We consider the problem of constructing a penalty function associated with a given monotone function. We provide a construction that allows the monotone function to be discontinuous or bounded. We show that, although the penalty may be non-convex, it is weakly convex. We briefly discuss the implications for iterative solutions of linear inverse problems.

Regularization of linear inverse problems with total generalized variation

Abstract The regularization properties of the total generalized variation (TGV) functional for the solution of linear inverse problems by means of Tikhonov regularization are studied. Considering the associated minimization problem for general symmetric tensor fields, the well-posedness is established in the space of symmetric tensor fields of bounded deformation, a generalization of the space of functions of bounded variation. Convergence for vanishing noise level is shown in a multiple regularization parameter framework in terms of the naturally arising notion of TGV-strict convergence. Finally, some basic properties, in particular non-equivalence for different parameters, are discussed for this notion.

Linear inverse problems and state estimation: Regularization, Observability and Convergence

Solving tomographic inverse problems in real-time is a challenging task due to the dimension of the problem, the nonlinearities involved and the fact that they are ill-posed problems. Efficient algorithms are required to address the application of tomographic technologies in the process industries. Several authors have proposed to address this issue by augmenting the measurement vector with fictitious outputs in order to solve the problem using state estimators. This paper analyses the solution of linear inverse problems from the perspective of a state estimation problem, providing links between regularization, observability and convergence of the algorithms. In addition, it also proposes an alternative way to define the fictitious outputs, leading to observers with fewer parameters than the one widely used in the literature. The simulation of a simple example illustrates the main ideas of this work.

Recovery Algorithms for Vector-Valued Data with Joint Sparsity Constraints

Vector-valued data appearing in concrete applications often possess sparse expansions with respect to a preassigned frame for each vector component individually. Additionally, different components may also exhibit common sparsity patterns. Recently, there were introduced sparsity measures that take into account such joint sparsity patterns, promoting coupling of nonvanishing components. These measures are typically constructed as weighted $\ell_1$ norms of componentwise $\ell_q$ norms of frame coefficients. We show how to compute solutions of linear inverse problems with such joint sparsity regularization constraints by fast thresholded Landweber algorithms. Next we discuss the adaptive choice of suitable weights appearing in the definition of sparsity measures. The weights are interpreted as indicators of the sparsity pattern and are iteratively updated after each new application of the thresholded Landweber algorithm. The resulting two-step algorithm is interpreted as a double-minimization scheme for a suitable target functional. We show its $\ell_2$-norm convergence. An implementable version of the algorithm is also formulated, and its norm convergence is proven. Numerical experiments in color image restoration are presented.

Convergence results for the Bayesian inversion theory

In this paper we derive convergence results for regularized solutions of linear inverse problems obtained by the Bayesian approach in the Ky Fan metric. We show that the convergence rate is order optimal in finite dimensional spaces. Moreover, we prove that order optimal rates can be obtained for weighted Bayesian solutions when the dimension goes to infinity.

An optimal control problem in image processing

AbstractAs a starting point, we present a control problem in mammographic image processing which leads to non‐standard penalty terms and involves a degenerate parabolic PDE which has to be controlled in the coefficients. We then discuss the classical conditional gradient method from constrained optimization and propose a generalization for non‐convex functionals which covers the conditional gradient method as well as the recently proposed iterative shrinkage method of Daubechies, Defrise and De Mol for the solution of linear inverse problems with sparsity promoting penalty terms. We prove that this new algorithm converges. This also gives a deeper understanding of the iterative shrinkage method. Further, we show an application to the above‐mentioned control problem in image processing. (© 2006 WILEY‐VCH Verlag GmbH &amp; Co. KGaA, Weinheim)

Delineation of layer boundaries from smooth models obtained from the geoelectrical sounding data inversion

Two techniques have been presented for the delineation of boundaries from smooth models obtained by smooth inversion techniques of geoelectrical sounding data, such as straightforward inversion scheme, Occam’s and Zohdy’s methods. The smooth model consists of a large number of equally spaced layers, wherein the real geological boundaries are missing. The techniques proposed here suppress the geologically irrelevant boundaries and support the real structural boundaries present in the geoelectrical data. In the first technique, solution of linear inverse problem is improved iteratively through weighted minimum norm inverse, the weight being taken from the current solution. The technique is referred as Iterative Straightforward Inversion Scheme. The second method is analytical, based on the application of smoothing filter, referred in the literature as edge-preserving smoothing. A few examples of theoretical magnetotelluric, dc resistivity and field sounding data have been presented to demonstrate the capabilities of the techniques. The methodologies also reduce the conspicuous oscillations in the smooth solutions caused by the conversion of sharp boundaries to the smooth ones.

Resolution of linear magnetostatic inverse problem using iterative regularization

This paper deals with the solution of linear inverse problems in magnetostatics. The case the authors have broached is finding the current density on the basis of magnetic field values. Solving this kind of equation is an ill-posed problem. Exact magnetic field values and measured values lead to different cases, each of which is presented. To solve them, the authors use the conjugate gradient method with iterative regularization. They present numerical results for the design of magnets, gradient and shim coils, and numerical results for the problem of recovering current density values from measured field values.

Unification of some deterministic and probabilistic methods for the solution of linear inverse problems via the principle of maximum entropy on the mean

This paper deals with the resolution of linear inverse problems. We present a mathematical study of the principle of maximum entropy on the mean (PMEM), and show that it is possible to derive from this principle many deterministic and probabilistic regularization techniques, including the well known Tikhonov method as well as the classical entropy method. In particular, within the deterministic family, full attention is devoted to the regularization principle of WIPE, a methodology recently introduced in radio imaging and optical interferometry. The principle in question is based on the concept of resolution as it is usually introduced in physics. The infinite-dimensional linearly constrained optimization problem underlying the PMEM is solved by means of a dual strategy: recent developments on partially finite convex programming are applied to our specific problem. To illustrate our analysis, we also present a few numerical simulations, in which the regularizer of WIPE is compared with entropy.