Tikhonov Problem Research Articles

Projected Newton method for noise constrained Tikhonov regularization

Tikhonov regularization is a popular approach to obtain a meaningful solution for ill-conditioned linear least squares problems. A relatively simple way of choosing a good regularization parameter is given by Morozov’s discrepancy principle. However, most approaches require the solution of the Tikhonov problem for many different values of the regularization parameter, which is computationally demanding for large scale problems. We propose a new and efficient algorithm which simultaneously solves the Tikhonov problem and finds the corresponding regularization parameter such that the discrepancy principle is satisfied. We achieve this by formulating the problem as a nonlinear system of equations and solving this system using a line search method. We obtain a good search direction by projecting the problem onto a low dimensional Krylov subspace and computing the Newton direction for the projected problem. This projected Newton direction, which is significantly less computationally expensive to calculate than the true Newton direction, is then combined with a backtracking line search to obtain a globally convergent algorithm, which we refer to as the projected Newton method. We prove convergence of the algorithm and illustrate the improved performance over current state-of-the-art solvers with some numerical experiments.

Sampled Tikhonov regularization for large linear inverse problems

In this paper we investigate iterative methods that are based on sampling of the data for computing Tikhonov-regularized solutions. We focus on very large inverse problems where access to the entire data set is not possible all-at-once (e.g. for problems with streaming or massive datasets). Row-access methods provide an ideal framework for solving such problems since they only require access to ‘blocks’ of the data at any given time. However, when using these iterative sampling methods to solve inverse problems, the main challenges include a proper choice of the regularization parameter, appropriate sampling strategies, and a convergence analysis. To address these challenges, we describe a family of sampled iterative methods that can incorporate data as they become available (e.g. randomly sampled). We consider two sampled iterative methods where the iterates can be characterized as solutions to a sequence of approximate Tikhonov problems. The first method requires the regularization parameter to be fixed a priori and converges asymptotically to an unregularized solution for randomly sampled data. This is undesirable for inverse problems. Thus, we focus on the second method where the main benefits are that the regularization parameter can be updated during the iterative process and the iterates converge asymptotically to a Tikhonov-regularized solution. We describe adaptive approaches to update the regularization parameter that are based on sampled residuals, and we provide a limited-memory variant for larger problems. Numerical examples, including a large-scale super-resolution imaging example, demonstrate the potential for these methods.

Learning regularization parameters for general-form Tikhonov

Computing regularization parameters for general-form Tikhonov regularization can be an expensive and difficult task, especially if multiple parameters or many solutions need to be computed in real time. In this work, we assume training data is available and describe an efficient learning approach for computing regularization parameters that can be used for a large set of problems. We consider an empirical Bayes risk minimization framework for finding regularization parameters that minimize average errors for the training data. We first extend methods from Chung et al (2011 SIAM J. Sci. Comput. 33 3132–52) to the general-form Tikhonov problem. Then we develop a learning approach for multi-parameter Tikhonov problems, for the case where all involved matrices are simultaneously diagonalizable. For problems where this is not the case, we describe an approach to compute near-optimal regularization parameters by using operator approximations for the original problem. Finally, we propose a new class of regularizing filters, where solutions correspond to multi-parameter Tikhonov solutions, that requires less data than previously proposed optimal error filters, avoids the generalized SVD, and allows flexibility and novelty in the choice of regularization matrices. Numerical results for 1D and 2D examples using different norms on the errors show the effectiveness of our methods.

Generalized Hybrid Iterative Methods for Large-Scale Bayesian Inverse Problems

We develop a generalized hybrid iterative approach for computing solutions to large-scale Bayesian inverse problems. We consider a hybrid algorithm based on the generalized Golub--Kahan bidiagonalization for computing Tikhonov regularized solutions to problems where explicit computation of the square root and inverse of the covariance kernel for the prior covariance matrix is not feasible. This is useful for large-scale problems where covariance kernels are defined on irregular grids or are available only via matrix-vector multiplication, e.g., those from the Matérn class. We show that iterates are equivalent to LSQR iterates applied to a directly regularized Tikhonov problem, after a transformation of variables, and we provide connections to a generalized singular value decomposition filtered solution. Our approach shares many benefits of standard hybrid methods such as avoiding semiconvergence and automatically estimating the regularization parameter. Numerical examples from image processing demonstrate the effectiveness of the described approaches.

Dynamic multi-source X-ray tomography using a spacetime level set method

A novel variant of the level set method is introduced for dynamic X-ray tomography. The target is allowed to change in time while being imaged by one or several source–detector pairs at a relatively high frame-rate. The algorithmic approach is motivated by the results in [22], showing that the modified level set method can tolerate highly incomplete projection data in stationary tomography. Furthermore, defining the level set function in spacetime enforces temporal continuity in the dynamic tomography context considered here. The tomographic reconstruction is found as a minimizer of a nonlinear functional. The functional contains a regularization term penalizing the L2 norms of up to n derivatives of the reconstruction. The case n=1 is shown to be equivalent to a convex Tikhonov problem that has a unique minimizer. For n≥2 the existence of a minimizer is proved under certain assumptions on the signal-to-noise ratio and the size of the regularization parameter. Numerical examples with both simulated and measured dynamic X-ray data are included, and the proposed method is found to yield reconstructions superior to standard methods such as FBP or non-negativity constrained Tikhonov regularization and favorably comparable to those of total variation regularization. Furthermore, the methodology can be adapted to a wide range of measurement arrangements with one or more X-ray sources.

A Hybrid LSMR Algorithm for Large-Scale Tikhonov Regularization

We develop a hybrid iterative approach for computing solutions to large-scale ill-posed inverse problems via Tikhonov regularization. We consider a hybrid LSMR algorithm, where Tikhonov regularization is applied to the LSMR subproblem rather than the original problem. We show that, contrary to standard hybrid methods, hybrid LSMR iterates are not equivalent to LSMR iterates on the directly regularized Tikhonov problem. Instead, hybrid LSMR leads to a different Krylov subspace problem. We show that hybrid LSMR shares many of the benefits of standard hybrid methods such as avoiding semiconvergence behavior. In addition, since the regularization parameter can be estimated during the iterative process, it does not need to be estimated a priori, making this approach attractive for large-scale problems. We consider various methods for selecting regularization parameters and discuss stopping criteria for hybrid LSMR, and we present results from image processing.

Volume Continuation of potential fields from the minimum-length solution: An optimal tool for continuation through general surfaces

Many methods have been used to upward continue potential field data. Most techniques employ the Fast Fourier transform, which is an accurate, quick way to compute level-to-level upward continuation or spatially varying scale filters for level-to-draped surfaces. We here propose a new continuation approach based on the minimum-length solution of the inverse potential field problem, which we call Volume Continuation (VOCO). For real data the VOCO is obtained as the regularized solution to the Tikhonov problem. We tested our method on several synthetic examples involving all types of upward continuation and downward continuation (level-to-level, level-to-draped, draped-to-level, draped-to-draped). We also employed the technique to upward continue to a constant height (2500ma.s.l.), the high-resolution draped aeromagnetic data of the Ischia Island in Southern Italy. We found that, on the average, they are consistent with the aeromagnetic regional data measured at the same altitude. The main feature of our method is that it does not only provide continued data over a specified surface, but it yields a volume of upward continuation. For example, the continued data refers to a volume and thus, any surface may be easily picked up within the volume to get upward continuation to different surfaces. This approach, based on inversion of the measured data, tends to be especially advantageous over the classical techniques when dealing with draped-to-level upward continuation. It is also useful to obtain a more stable downward continuation and to continue noisy data. The inversion procedure involved in the method implies moderate computational costs, which are well compensated by getting a 3D set of upward continued data to achieve high quality results.

Simple and Efficient Determination of the Tikhonov Regularization Parameter Chosen by the Generalized Discrepancy Principle for Discrete Ill-Posed Problems

Discrete ill-posed problems where both the coefficient matrix and the right hand side are contaminated by noise appear in a variety of engineering applications. In this paper we consider Tikhonov regularized solutions where the regularization parameter is chosen by the generalized discrepancy principle (GDP). In contrast to Newton-based methods often used to compute such parameter, we propose a new algorithm referred to as GDP-FP, where derivatives are not required and where the regularization parameter is calculated efficiently by a fixed-point iteration procedure. The algorithm is globally and monotonically convergent. Additionally, a specialized version of GDP-FP based on a projection method, that is well-suited for large-scale Tikhonov problems, is also proposed and analyzed in detail. Numerical examples are presented to illustrate the effectiveness of the proposed algorithms on test problems from the literature.

Infinite-σ Limits For Tikhonov Regularization

We consider the problem of Tikhonov regularization with a general convex loss function: this formalism includes support vector machines and regularized least squares. For a family of kernels that includes the Gaussian, parameterized by a bandwidth parameter σ, we characterize the limiting solution as σ → ∞. In particular, we show that if we set the regularization parameter λ = ~λ σ-2p, the regularization term of the Tikhonov problem tends to an indicator function on polynomials of degree ⌊p⌋ (with residual regularization in the case where p ∈ Z). The proof rests on two key ideas: epi-convergence, a notion of functional convergence under which limits of minimizers converge to minimizers of limits, and a value-based formulation of learning, where we work with regularization on the function output values (y) as opposed to the function expansion coefficients in the RKHS. Our result generalizes and unifies previous results in this area.

Some topics on weighted Moore–Penrose inverse, weighted least squares and weighted regularized Tikhonov problems

It is well known that the normwise relative condition numbers measure the sensitivity of matrix inversion and the solution of linear systems. The classical normwise relative condition number formulas are given by Higham [Linear Algebra Appl. 214 (1995) 193]. Here, we consider the condition number formulas for the weighted Moore–Penrose inverse of a rectangular matrix and give explicit expressions for the weighted condition numbers of the singular linear systems Ax= b. Moreover, we consider the nearness to rank deficiency. Finally, we give some decomposition tools and their application in the regularized weighted Tikhonov problems.

Perturbation Identities for Regularized Tikhonov Inverses and Weighted Pseudoinverses

We consider the perturbation analysis of two important problems for solving ill-conditioned or rank-deficient linear least squares problems. The Tikhonov regularized problem is a linear least squares problem with a regularization term balancing the size of the residual against the size of the weighted solution. The weight matrix can be a non-square matrix (usually with fewer rows than columns). The minimum-norm problem is the minimization of the size of the weighted solutions given by the set of solutions to the, possibly rank-deficient, linear least squares problem.It is well known that the solution of the Tikhonov problem tends to the minimum-norm solution as the regularization parameter of the Tikhonov problem tends to zero. Using this fact and the generalized singular value decomposition enable us to make a perturbation analysis of the minimum-norm problem with perturbation results for the Tikhonov problem. From the analysis we attain perturbation identities for Tikhonov inverses and weighted pseudoinverses.