Regularized Minimization Problem Research Articles

Stability and error estimates of BV solutions to the Abel inverse problem

Reconstructing images from ill-posed inverse problems often utilizes total variation regularization in order to recover discontinuities in the data while also removing noise and other artifacts. Total variation regularization has been successful in recovering images for (noisy) Abel transformed data, where object boundaries and data support will lead to sharp edges in the reconstructed image. In this work, we analyze the behavior of solutions to the Abel inverse problem, deriving a priori estimates on the recovery. In particular, we provide L2-stability bounds on solutions to the Abel inverse problem. These bounds yield error estimates on images reconstructed from a proposed total variation regularized minimization problem.

Equivalent Lipschitz surrogates for zero-norm and rank optimization problems

This paper proposes a mechanism to produce equivalent Lipschitz surrogates for zero-norm and rank optimization problems by means of the global exact penalty for their equivalent mathematical programs with an equilibrium constraint (MPECs). Specifically, we reformulate these combinatorial problems as equivalent MPECs by the variational characterization of the zero-norm and rank function, show that their penalized problems, yielded by moving the equilibrium constraint into the objective, are the global exact penalization, and obtain the equivalent Lipschitz surrogates by eliminating the dual variable in the global exact penalty. These surrogates, including the popular SCAD function in statistics, are also difference of two convex functions (D.C.) if the function and constraint set involved in zero-norm and rank optimization problems are convex. We illustrate an application by designing a multi-stage convex relaxation approach to the rank plus zero-norm regularized minimization problem.

A joint matrix minimization approach for multi-image face recognition

The Schatten p-quasi-norm regularized minimization problem has attracted extensive attention in machine learning, image recognition, signal reconstruction, etc. Meanwhile, the l2,1-regularized matrix optimization models are also popularly used for its joint sparsity. Naturally, the pseudo matrix norm l2,p is expected to carry over the advantages of both l p and l2,1. This paper proposes a mixed l2,q-l2,p matrix minimization approach for multi-image face recognition. To uniformly solve this optimization problem for any q ∈ [1, 2] and p ∈ (0, 2], an iterative quadratic method (IQM) is developed. IQM is proved to descend strictly until it gets a stationary point of the mixed l2,q-l2,p matrix minimization. Moreover, a more practical IQM is presented for large-scale case. Experimental results on three public facial image databases show that the joint matrix minimization approach with practical IQM not only saves much computational cost but also achieves better performance in face recognition than state-of-the-art methods.

Global optimality condition and fixed point continuation algorithm for non-Lipschitz ℓ p regularized matrix minimization

Regularized minimization problems with nonconvex, nonsmooth, even non-Lipschitz penalty functions have attracted much attention in recent years, owing to their wide applications in statistics, control, system identification and machine learning. In this paper, the non-Lipschitz $\ell_p~(0<p<1)$ regularized matrix minimization problem is studied. A global necessary optimality condition for this non-Lipschitz optimization problem is firstly obtained, specifically, the global optimal solutions for the problem are fixed points of the so-called $p$-thresholding operator which is matrix-valued and set-valued. Then a fixed point iterative scheme for the non-Lipschitz model is proposed, and the convergence analysis is also addressed in detail. Moreover, some acceleration techniques are adopted to improve the performance of this algorithm. The effectiveness of the proposed $p$-thresholding fixed point continuation ($p$-FPC) algorithm is demonstrated by numerical experiments on randomly generated and real matrix completion problems.

Inverse problems with nonnegative and sparse solutions: algorithms and application to the phase retrieval problem

In this paper, we study a gradient-type method and a semismooth Newton method for minimization problems in regularizing inverse problems with nonnegative and sparse solutions. We propose a special penalty functional forcing the minimizers of regularized minimization problems to be nonnegative and sparse, and then we apply the proposed algorithms in a practical the problem. The strong convergence of the gradient-type method and the local superlinear convergence of the semismooth Newton method are proven. Then, we use these algorithms for the phase retrieval problem and illustrate their efficiency in numerical examples, particularly in the practical problem of optical imaging through scattering media where all the noises from experiment are presented.

Contingent derivatives and regularization for noncoercive inverse problems

We study the inverse problem of parameter identification in noncoercive variational problems that commonly appear in applied models. We examine the differentiability of the set-valued parameter-to-solution map using the first-order and the second-order contingent derivatives. We explore the inverse problem using the output least-squares and the modified output least-squares objectives. By regularizing the noncoercive variational problem, we obtain a single-valued regularized parameter-to-solution map and investigate its smoothness and boundedness. We also consider optimization problems using the output least-squares and the modified output least-squares objectives for the regularized variational problem. We give a complete convergence analysis showing that for the output least-squares and the modified output least-squares, the regularized minimization problems approximate the original optimization problems suitably. We also provide the first-order and the second-order adjoint method for the computation of the first-order and the second-order derivatives of the output least-squares objective. We provide discrete formulas for the gradient and the Hessian calculation and present numerical results.

Spatial-temporal representatives selection and weighted patch descriptor for person re-identification

Abstract How to represent the sequential person images is a crucial issue in multi-shot person re-identification. In this paper, we propose to select the spatial-temporal informative representatives to describe the image sequence. Specifically, we address representatives selection as a row-sparsity regularized minimization problem which can be effectively solved via convex programming. The sparsity of the representatives is controlled by a regularization parameter based on both spatial and temporal dissimilarities. Furthermore, we design a weighted patch descriptor by employing the random walk with restart model to propagate the patch weights on the person image. Finally, we utilize the cross-view quadratic discriminant analysis as the metric learning to mitigate the cross-view gaps among different cameras. Extensive experiments on three benchmark datasets iLIDS-VID, PRID 2011 and SAIVT-SoftBio demonstrate the promising performance of the proposed method.

A pseudo-heuristic parameter selection rule forl1-regularized minimization problems

This paper considers the regularization parameter determination of l 1 -regularized minimization problem. We solve the l 1 -regularized problem using iterative reweighted least squares (IRLS) which involves solving a linear system whose coefficient matrix has the form α M + ( 1 − α ) N ( α ∈ ( 0 , 1 ) ). The aim of this paper is to find an efficient and computationally inexpensive algorithm to both choose the regularization parameter and solve the l 1 -regularized problem. In order to achieve this, we propose an IRLS algorithm with adaptive regularization parameter selection based on a heuristic parameter determination rule—de Boor’s parameter selection criterion. Compared with some of the state-of-the-art algorithms and parameter selection rules, the numerical experiments show the efficiency and robustness of the proposed method.

Two-stage convex relaxation approach to low-rank and sparsity regularized least squares loss

In this paper we consider the rank and zero norm regularized least squares loss minimization problem with a spectral norm ball constraint. For this class of NP-hard optimization problems, we propose a two-stage convex relaxation approach by majorizing some suitable locally Lipschitz continuous surrogates. Furthermore, the Frobenius norm error bound for the optimal solution of each stage is characterized and the theoretical guarantee is established for the two-stage convex relaxation approach by showing that the error bound of the first stage convex relaxation (i.e., the nuclear norm and $$\ell _1$$ -norm regularized minimization problem), can be reduced much by the second stage convex relaxation under a suitable restricted eigenvalue condition. Also, we verify the efficiency of the proposed approach by applying it to some random test problems and some real problems.

The MFS for the identification of a sound-soft interior acoustic scatterer

We employ the method of fundamental solutions (MFS) for detecting a sound-soft scatterer surrounding a host acoustic homogeneous medium due to a given point source inside it. The measurements are taken inside the medium and, in addition, are contaminated with noise. The MFS discretization yields a nonlinear constrained regularized minimization problem which is solved using standard software. The results of several numerical experiments are presented and discussed.

University of Miami School of Medicine to Become an Osher Center

We employ the method of fundamental solutions (MFS) for detecting a sound-soft scatterer surrounding a host acoustic homogeneous medium due to a given point source inside it. The measurements are taken inside the medium and, in addition, are contaminated with noise. The MFS discretization yields a nonlinear constrained regularized minimization problem which is solved using standard software. The results of several numerical experiments are presented and discussed.

Adaptive linearized alternating direction method of multipliers for non-convex compositely regularized optimization problems

We consider a wide range of non-convex regularized minimization problems, where the non-convex regularization term is composite with a linear function engaged in sparse learning. Recent theoretical investigations have demonstrated their superiority over their convex counterparts. The computational challenge lies in the fact that the proximal mapping associated with non-convex regularization is not easily obtained due to the imposed linear composition. Fortunately, the problem structure allows one to introduce an auxiliary variable and reformulate it as an optimization problem with linear constraints, which can be solved using the Linearized Alternating Direction Method of Multipliers (LADMM). Despite the success of LADMM in practice, it remains unknown whether LADMM is convergent in solving such non-convex compositely regularized optimizations. In this research, we first present a detailed convergence analysis of the LADMM algorithm for solving a non-convex compositely regularized optimization problem with a large class of non-convex penalties. Furthermore, we propose an Adaptive LADMM (AdaLADMM) algorithm with a line-search criterion. Experimental results on different genres of datasets validate the efficacy of the proposed algorithm.

A regularized tri-linear approach for optical interferometric imaging

In the context of optical interferometry, only undersampled power spectrum and bispectrum data are accessible. It poses an ill-posed inverse problem for image recovery. Recently, a tri-linear model was proposed for monochromatic imaging, leading to an alternated minimization problem. In that work, only a positivity constraint was considered, and the problem was solved by an approximated Gauss-Seidel method. In this paper, we propose to improve the approach on three fundamental aspects. Firstly, we define the estimated image as a solution of a regularized minimization problem, promoting sparsity in a fixed dictionary using either an $\ell_1$ or a weighted-$\ell_1$ regularization term. Secondly, we solve the resultant non-convex minimization problem using a block-coordinate forward-backward algorithm. This algorithm is able to deal both with smooth and non-smooth functions, and benefits from convergence guarantees even in a non-convex context. Finally, we generalize our model and algorithm to the hyperspectral case, promoting a joint sparsity prior through an $\ell_{2,1}$ regularization term. We present simulation results, both for monochromatic and hyperspectral cases, to validate the proposed approach.

Minimization of transformed $l_1$ penalty: Closed form representation and iterative thresholding algorithms

The transformed $l_1$ penalty (TL1) functions are a one parameter family of bilinear transformations composed with the absolute value function. When acting on vectors, the TL1 penalty interpolates $l_0$ and $l_1$ similar to $l_p$ norm ($p \in (0,1)$). In our companion paper, we showed that TL1 is a robust sparsity promoting penalty in compressed sensing (CS) problems for a broad range of incoherent and coherent sensing matrices. Here we develop an explicit fixed point representation for the TL1 regularized minimization problem. The TL1 thresholding functions are in closed form for all parameter values. In contrast, the $l_p$ thresholding functions ($p \in [0,1]$) are in closed form only for $p=0,1,1/2,2/3$, known as hard, soft, half, and 2/3 thresholding respectively. The TL1 threshold values differ in subcritical (supercritical) parameter regime where the TL1 threshold functions are continuous (discontinuous) similar to soft-thresholding (half-thresholding) functions. We propose TL1 iterative thresholding algorithms and compare them with hard and half thresholding algorithms in CS test problems. For both incoherent and coherent sensing matrices, a proposed TL1 iterative thresholding algorithm with adaptive subcritical and supercritical thresholds consistently performs the best in sparse signal recovery with and without measurement noise.

Multistage Convex Relaxation Approach to Rank Regularized Minimization Problems Based on Equivalent Mathematical Program with a Generalized Complementarity Constraint

This paper concerns itself with the rank regularized minimization problem with a spectral norm ball constraint. We reformulate this NP-hard problem as an equivalent MPGCC (mathematical program with a generalized complementarity constraint) and show that the penalty problem, yielded by moving the generalized complementarity constraint to the objective, is exact in the sense that it has the same global optimal solution set as the MPGCC does when the penalty parameter is over a threshold. Then, by solving the exact penalty problem in an alternating way, we propose a multistage convex relaxation approach, and provide its theoretical guarantee for the rank regularized least squares problem with a spectral norm ball constraint. Specifically, we derive the error and approximate rank bounds for the optimal solution of each stage and quantify the reduction of the error and approximate rank bounds of the first stage convex relaxation (which is exactly the nuclear norm relaxation) in the subsequent stages. In partic...

Textile porosity determination based on a nonlinear heat and moisture transfer model

The inverse problems of textile materials design on heat and moisture transfer properties are important and indispensable in applications in the body-clothing-environment system. We present an inverse problem of textile porosity determination (IPTPD) based on a nonlinear heat and moisture transfer model. Adopting the idea of the least-squares, the mathematical formulation of IPTPD is deduced to a regularized optimization problem with collocation method applied. The continuity of the regularized minimization problem is proved. By means of genetic algorithm (GA), the approximate solution of the IPTPD is numerically obtained. To reduce the computational cost, an improved algorithm based on BP neural network with GA is proposed in the numerical simulation. Compared with the direct GA searching, the computational cost is greatly reduced, which presents a similar result.

A MEMORY EFFICIENT INCREMENTAL GRADIENT METHOD FOR REGULARIZED MINIMIZATION

In this paper, we propose a new incremental gradient method for solving a regularized minimization problem whose objective is the sum of m smooth functions and a (possibly nonsmooth) convex function. This method uses an adaptive stepsize. Recently proposed incremental gradient methods for a regularized minimization problem need O(mn) storage, where n is the number of variables. This is the drawback of them. But, the proposed new incremental gradient method requires only O(n) storage.

Fast Nonsmooth Regularized Risk Minimization with Continuation

In regularized risk minimization, the associated optimization problem becomes particularly difficult when both the loss and regularizer are nonsmooth. Existing approaches either have slow or unclear convergence properties, are restricted to limited problem subclasses, or require careful setting of a smoothing parameter. In this paper, we propose a continuation algorithm that is applicable to a large class of nonsmooth regularized risk minimization problems, can be flexibly used with a number of existing solvers for the underlying smoothed subproblem, and with convergence results on the whole algorithm rather than just one of its subproblems. In particular, when accelerated solvers are used, the proposed algorithm achieves the fastest known rates of $O(1/T^2)$ on strongly convex problems, and $O(1/T)$ on general convex problems. Experiments on nonsmooth classification and regression tasks demonstrate that the proposed algorithm outperforms the state-of-the-art.

A Proximal Alternating Direction Method for Semi-Definite Rank Minimization

Semi-definite rank minimization problems model a wide range of applications in both signal processing and machine learning fields. This class of problem is NP-hard in general. In this paper, we propose a proximal Alternating Direction Method (ADM) for the well-known semi-definite rank regularized minimization problem. Specifically, we first reformulate this NP-hard problem as an equivalent biconvex MPEC (Mathematical Program with Equilibrium Constraints), and then solve it using proximal ADM, which involves solving a sequence of structured convex semi-definite subproblems to find a desirable solution to the original rank regularized optimization problem. Moreover, based on the Kurdyka-Lojasiewicz inequality, we prove that the proposed method always converges to a KKT stationary point under mild conditions. We apply the proposed method to the widely studied and popular sensor network localization problem. Our extensive experiments demonstrate that the proposed algorithm outperforms state-of-the-art low-rank semi-definite minimization algorithms in terms of solution quality.

A Splitting-based Iterative Method for Sparse Reconstruction

In this paper, we study a ℓ1-norm regularized minimization method for sparse solution recovery in compressed sensing and X-ray CT image reconstruction. In the proposed method, an alternating minimization algorithm is employed to solve the involved ℓ1-norm regularized minimization problem. Under some suitable conditions, the proposed algorithm is shown to be globally convergent. Numerical results indicate that the presented method is effective and promising.