Iterative Reweighted Least Squares Algorithm Research Articles

An accelerated non-Gaussianity based multichannel predictive deconvolution method with the limited supporting region of filters

The multichannel predictive deconvolution can be conducted in overlapping temporal and spatial data windows to solve the 2D predictive filter for multiple removal. Generally, the 2D predictive filter can better remove multiples at the cost of more computation time compared with the 1D predictive filter. In this paper we first use the cross-correlation strategy to determine the limited supporting region of filters where the coefficients play a major role for multiple removal in the filter coefficient space. To solve the 2D predictive filter the traditional multichannel predictive deconvolution uses the least squares (LS) algorithm, which requires primaries and multiples are orthogonal. To relax the orthogonality assumption the iterative reweighted least squares (IRLS) algorithm and the fast iterative shrinkage thresholding (FIST) algorithm have been used to solve the 2D predictive filter in the multichannel predictive deconvolution with the non-Gaussian maximization (L1 norm minimization) constraint of primaries. The FIST algorithm has been demonstrated as a faster alternative to the IRLS algorithm. In this paper we introduce the FIST algorithm to solve the filter coefficients in the limited supporting region of filters. Compared with the FIST based multichannel predictive deconvolution without the limited supporting region of filters the proposed method can reduce the computation burden effectively while achieving a similar accuracy. Additionally, the proposed method can better balance multiple removal and primary preservation than the traditional LS based multichannel predictive deconvolution and FIST based single channel predictive deconvolution. Synthetic and field data sets demonstrate the effectiveness of the proposed method.

Joint estimation of (co) variance components and breeding values for mean and dispersion of days from calving to first service in Holstein cow

The present study explored the possibility of selection for uniformity of days from calving to first service (DFS) in dairy cattle. A double hierarchical generalised linear model with an iterative reweighted least-squares algorithm was used to estimate covariance components for the mean and dispersion of DFS. Data included the records of 27 113 Iranian Holstein cows (parity, 1–6) in 15 herds from 1981 to 2007. The estimated additive genetic variance for the mean and dispersion were 32.25 and 0.0139; both of these values had low standard errors. The genetic standard deviation for dispersion of DFS was 0.117, indicating that decreasing the estimated breeding value of dispersion by one genetic standard deviation can increase the uniformity by 12%. A strong positive genetic correlation (0.689) was obtained between the mean and dispersion of DFS. This genetic correlation is favourable since one of the aims of breeding is to simultaneously decrease the mean and increase the uniformity of DFS. The Spearman rank correlations between estimated breeding values in the mean and dispersion for sires with a different number of daughter observations were 0.907. In the studied population, the genetic trend in the mean of DFS was significant and favourable (–0.063 days/year), but the genetic trend in the dispersion of DFS was not significantly different from zero. The results obtained in the present study indicated that the mean and uniformity of DFS can simultaneously be improved in dairy cows.

An iterative re-weighted least-squares algorithm for the design of active abso- rbing wavemaker controller

In physical model tests for highly reflective structures, one often encounters a problem of multiple reflections between the reflective structures and the wavemaker. Absorbing wavemakers can cancel the re-reflective waves by adjusting the paddle motion. In this paper, we propose a method to design the controller of the 2-D absorbing wavemaker system in the wave flume. Based on the first-order wavemaker theory, a frequency domain absorption transfer function is derived. Its time realization can be obtained by de- signing an infinite impulse response (IIR) digital filter, which is expected to approximate the absorption transfer function in the least- squares sense. A commonly used approach to determine the parameters of the IIR filter is applying the Taylor expansion to linearize the filter formulation and solving the linear least-squares problem. However, the result is not optimal because the linearization cha- nges the original objective function. To improve the approximation performance, we propose an iterative reweighted least-squares (IRLS) algorithm and demonstrate that with the filters designed by this algorithm, the approximation errors can be reduced. Physical experiments are carried out with the designed controller. The results show that the system performs well for both regular and irregu- lar waves.

Non-uniform sampling of NMR relaxation data.

The use of non-uniform sampling of NMR spectra may give significant reductions in the data acquisition time. For quantitative experiments such as the measurement of spin relaxation rates, non-uniform sampling is however not widely used as inaccuracies in peak intensities may lead to errors in the extracted dynamic parameters. By systematic reducing the coverage of the Nyquist grid of (15)N Carr-Purcell-Meiboom-Gill (CPMG) relaxation dispersion datasets for four different proteins and performing a full data analysis of the resulting non-uniform sampled datasets, we have compared the performance of the multi-dimensional decomposition and iterative re-weighted least-squares algorithms in reconstructing spectra with accurate peak intensities. As long as a single fully sampled spectrum is included in a series of otherwise non-uniform sampled two-dimensional spectra, multi-dimensional decomposition reconstructs the non-uniform sampled spectra with high accuracy. For two of the four analyzed datasets, a coverage of only 20% results in essentially the same results as the fully sampled data. As exemplified by other data, such a low coverage is in general not enough to produce reliable results. We find that a coverage level not compromising the final results can be estimated by recording a single full two-dimensional spectrum and reducing the spectrum quality in silico.

Multichannel predictive deconvolution based on the fast iterative shrinkage-thresholding algorithm

Multichannel predictive deconvolution can accommodate the lateral variations of subsurface structures to some extent and better preserve primaries than can single-channel predictive deconvolution. To solve the 2D predictive filter, traditional multichannel predictive deconvolution uses the least-squares (LS) algorithm, which requires orthogonality between primaries and multiples. In areas where primaries and multiples overlap, traditional LS-based multichannel predictive deconvolution can cause distorted primaries and residual multiples. To avoid the orthogonality assumption required by the LS algorithm, the iterative reweighted LS (IRLS) algorithm and the fast iterative shrinkage-thresholding (FIST) algorithm can be used to solve the prediction filter using the [Formula: see text] norm. The FIST algorithm uses the shrinkage-thresholding operator to promote the sparsity of estimated primaries and solves the predictive filter with iterative steps. Compared with the IRLS algorithm, the FIST algorithm can reduce the computation burden effectively while achieving similar accuracy. We have used the FIST algorithm for multichannel predictive deconvolution using the [Formula: see text] norm. Compared with traditional FIST-based single-channel predictive deconvolution and LS-based multichannel predictive deconvolution, our method can better balance primary preservation and multiple removal. Tests using synthetic and field data sets proved the effectiveness of the proposed method.

Complex-valued sparse recovery via double-threshold sigmoid penalty

The thresholding methods based on the generalized iteratively reweighted least squares (IRLS) iteration are discussed under the complex-valued condition in this paper. A new thresholding function (Double-Threshold Sigmoid (DTHS) function) and two associated algorithms (DTHS-1 and DTHS-2) are proposed herein, and their convergence performances are discussed in detail. It is shown that the generalized IRLS algorithm is unbiased if the thresholding penalty can eliminate the undesired perturbation term added on the correlation matrix of the measurement matrix. Compared with the others, the new algorithms are endowed with stability and insensitivity with respect to the regularization parameter by selecting some sound upper thresholds and dividing the iteration procedures into the degraded stage and DTHS stage respectively. Further analyses show that the DTHS-1 algorithm is suitable to deal with the sparse and continuous problems for both of the i.i.d. random matrix and under-resolution PSF matrix. The noise performance of the DTHS-1 algorithm is always superior to that of the IRLS algorithm, especially in the face of the under-resolution PSF matrix.

An Inner-Product-Based Discriminative IRLS Algorithm for Sparse Hyperspectral Detection

Automatic target detection is an important application in the hyperspectral image processing field. It is currently known that any test pixels in hyperspectral images can be represented within a spectral dictionary using appropriate sparse coefficients. Based on this assumption, some sparsity-based algorithms are developed for hyperspectral detection. This kind of sparse learning method attempts to find the sparse representation from a spectral library, i.e., a dictionary data set from which useful information is extracted. Among these algorithms, the iteratively reweighted least squares (IRLS) strategy is believed to be a simple and useful tool for sparse representation. However, when dealing with the hyperspectral data, the dictionary for sparse learning is usually high-dimensional which dramatically increases the scale and complexity of sparse learning. In such cases, most sparsitybased algorithms including the IRLS strategy lost their efficacy. To deal with this situation, we propose a discriminative IRLS algorithm, called inner-product-based discriminative IRLS detector (IDIRLSD), which decreases the scale and complexity problem by discriminatively seeking a subdictionary that retains the most critical information. Also, IDIRLSD applies a convex minimization for approximately solving the sparse recovery problem. A weighted l 1 minimization is relaxed and solved by IRLS strategy. The proposed algorithm applies an inner-product-based function for constructing the small-scale weighted l 1 minimization with respect to the subdictionary. The solution provided by IDIRLSD is then applied to label the test pixel as target or background. Experimental results from both synthetic and real hyperspectral data demonstrate the improved efficacy of the proposed algorithm.

Feature Matching with Bounded Distortion

We consider the problem of finding a geometrically consistent set of point matches between two images. We assume that local descriptors have provided a set of candidate matches, which may include many outliers. We then seek the largest subset of these correspondences that can be aligned perfectly using a nonrigid deformation that exerts a bounded distortion. We formulate this as a constrained optimization problem and solve it using a constrained, iterative reweighted least-squares algorithm. In each iteration of this algorithm we solve a convex quadratic program obtaining a globally optimal match over a subset of the bounded distortion transformations. We further prove that a sequence of such iterations converges monotonically to a critical point of our objective function. We show experimentally that this algorithm produces excellent results on a number of test sets, in comparison to several state-of-the-art approaches.

Adaptive multiple subtraction based on 3D blind separation of convolved mixtures

Multiple removal is one of the important preprocessing steps in a seismic data processing sequence. For an accurate multiple prediction, a full 3D method is required. However, the application of such methods is often limited by practical and economic constraints. Therefore, in practical situations, 2D prediction methods are used, such as 2D surface-related multiple elimination. However, the resulting predicted multiples may have significant temporal shift, spatial mismatch, and amplitude inconsistency, compared with true 3D multiples. Adaptive multiple subtraction based on 2D blind separation of convolved mixtures (BSCM) has been proposed to estimate the 2D matching filter in a single gather. To improve the flexibility of the adaptive multiple subtraction for the inconsistencies between the 2D predicted multiples and true 3D multiples, we evaluated the adaptive multiple subtraction as a problem of 3D BSCM. In the proposed method, the predicted multiples were modeled as the convolution of the true multiples with a 3D kernel, whose third dimension is in the gather direction. By maximizing the non-Gaussianity of the estimated primaries, the iterative reweighted least-squares algorithm was exploited to obtain the 3D matching filter, which is the inverse of the 3D kernel. To avoid the possible overfitting problem introduced by the 3D matching filter, the proposed method fit several seismic gathers using one 3D matching filter. In addition, by using the non-Gaussian maximization criterion, the proposed method alleviated the orthogonality assumption used by the least-squares subtraction method. Furthermore, the proposed method can eliminate the temporal and spatial mismatches between the 2D predicted multiples and true 3D multiples better than the 2D BSCM subtraction method. Tests on synthetic and field data sets demonstrated the effectiveness of the 3D BSCM subtraction method.

On the Convergence of the IRLS Algorithm in Non-Local Patch Regression

Recently, it was demonstrated in [CS2012,CS2013] that the robustness of the classical Non-Local Means (NLM) algorithm [BCM2005] can be improved by incorporating $\ell^p (0 < p \leq 2)$ regression into the NLM framework. This general optimization framework, called Non-Local Patch Regression (NLPR), contains NLM as a special case. Denoising results on synthetic and natural images show that NLPR consistently performs better than NLM beyond a moderate noise level, and significantly so when $p$ is close to zero. An iteratively reweighted least-squares (IRLS) algorithm was proposed for solving the regression problem in NLPR, where the NLM output was used to initialize the iterations. Based on exhaustive numerical experiments, we observe that the IRLS algorithm is globally convergent (for arbitrary initialization) in the convex regime $1 \leq p \leq 2$, and locally convergent (fails very rarely using NLM initialization) in the non-convex regime $0 < p < 1$. In this letter, we adapt the "majorize-minimize" framework introduced in [Voss1980] to explain these observations. [CS2012] Chaudhury et al. (2012), "Non-local Euclidean medians," IEEE Signal Processing Letters. [CS2013] Chaudhury et al. (2013), "Non-local patch regression: Robust image denoising in patch space," IEEE ICASSP. [BCM2005] Buades et al. (2005), "A review of image denoising algorithms, with a new one," Multiscale Modeling and Simulation. [Voss1980] Voss et al. (1980), "Linear convergence of generalized Weiszfeld's method," Computing.

On recovery of block-sparse signals via mixed l 2 /l q (0 &lt; q ≤ 1) norm minimization

Compressed sensing (CS) states that a sparse signal can exactly be recovered from very few linear measurements. While in many applications, real-world signals also exhibit additional structures aside from standard sparsity. The typical example is the so-called block-sparse signals whose non-zero coefficients occur in a few blocks. In this article, we investigate the mixed l 2/l q (0 < q ≤ 1) norm minimization method for the exact and robust recovery of such block-sparse signals. We mainly show that the non-convex l 2/l q (0 < q < 1) minimization method has stronger sparsity promoting ability than the commonly used l 2/l 1 minimization method both practically and theoretically. In terms of a block variant of the restricted isometry property of measurement matrix, we present weaker sufficient conditions for exact and robust block-sparse signal recovery than those known for l 2/l 1 minimization. We also propose an efficient Iteratively Reweighted Least-Squares (IRLS) algorithm for the induced non-convex optimization problem. The obtained weaker conditions and the proposed IRLS algorithm are tested and compared with the mixed l 2/l 1 minimization method and the standard l q minimization method on a series of noiseless and noisy block-sparse signals. All the comparisons demonstrate the outperformance of the mixed l 2/l q (0 < q < 1) method for block-sparse signal recovery applications, and meaningfulness in the development of new CS technology.

Minimizing Nonconvex Functions for Sparse Vector Reconstruction

In this paper, we develop a novel methodology for minimizing a class of nonconvex (concave on the non-negative orthant) functions for solving an underdetermined linear system of equations As = x when the solution vector s is known a priori to be sparse. The proposed technique is based on locally replacing the original objective function by a quadratic convex function which is easily minimized. The resulting algorithm is iterative and is absolutely converging to a fixed point of the original objective function. For a certain selection of convex objective functions, the class of algorithms called iterative reweighted least squares (IRLS) is shown to be a special case of the proposed methodology. Thus, the proposed algorithms are a generalization and unification of the previous methods. In addition, we also propose a new class of algorithms with better convergence properties compared to the regular IRLS algorithms and, hence, can be considered as enhancements to these algorithms. Since the original objective functions are nonconvex, the proposed algorithm is susceptible to convergence to a local minimum. To alleviate this difficulty, we propose a random perturbation technique that enhances the performance of the proposed algorithm. The numerical results show that the proposed algorithms outperform some of the well-known algorithms that are usually utilized for solving the same problem.

Sparse Signal Reconstruction via Iterative Support Detection

We present a novel sparse signal reconstruction method, iterative support detection (ISD), aiming to achieve fast reconstruction and a reduced requirement on the number of measurements compared to the classical $\ell_1$ minimization approach. ISD addresses failed reconstructions of $\ell_1$ minimization due to insufficient measurements. It estimates a support set I from a current reconstruction and obtains a new reconstruction by solving the minimization problem $\min\{\sum_{i\notin I}|x_i|:Ax=b\}$, and it iterates these two steps for a small number of times. ISD differs from the orthogonal matching pursuit method, as well as its variants, because (i) the index set I in ISD is not necessarily nested or increasing, and (ii) the minimization problem above updates all the components of x at the same time. We generalize the null space property to the truncated null space property and present our analysis of ISD based on the latter. We introduce an efficient implementation of ISD, called threshold-ISD, for recovering signals with fast decaying distributions of nonzeros from compressive sensing measurements. Numerical experiments show that threshold-ISD has significant advantages over the classical $\ell_1$ minimization approach, as well as two state-of-the-art algorithms: the iterative reweighted $\ell_1$ minimization algorithm (IRL1) and the iterative reweighted least-squares algorithm (IRLS). MATLAB code is available for download from http://www.caam.rice.edu/ optimization/L1/ISD/.

Optimization of two-dimensional IIR filters with nonseparable and separable denominator

We present algorithms for the optimization of two-dimensional (2-D) infinite impulse response (IIR) filters with separable or nonseparable denominator, for least squares or Chebyshev criteria. The algorithms are iterative, and each iteration consists of solving a semidefinite programming problem. For least squares designs, we adapt the Gauss-Newton idea, which outcomes to a convex approximation of the optimization criterion. For Chebyshev designs, we adapt the iterative reweighted least squares (IRLS) algorithm; in each iteration, a least squares Gauss-Newton step is performed, while the weights are changed as in the basic IRLS algorithm. The stability of the 2-D IIR filters is ensured by keeping the denominator inside convex stability domains, which are defined by linear matrix inequalities. For the 2-D (nonseparable) case, this is a new contribution, based on the parameterization of 2-D polynomials that are positive on the unit bicircle. In the experimental section, 2-D IIR filters with separable and nonseparable denominators are designed and compared. We show that each type may be better than the other, depending on the design specification. We also give an example of filter that is clearly better than a recent very good design.

Using the EM algorithm to train neural networks: misconceptions and a new algorithm for multiclass classification.

The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perceptron (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets.

An efficient iterative reweighted least-squares algorithm for two-dimensional FIR filters design in the Lp sense

This paper presents an efficient iterative reweighted least-squares (IRLS) algorithm to obtain an Lp approximation for the design of two-dimensional FIR filters. This algorithm introduces an extra frequency response which implicitly includes the weighting function such that the p-power error to be minimized can be represented in a twodimensional matrix form. The proposed algorithm reduces the computational complexity from O(N6) to O(N3), and storage space from O(N4) to O(N2), compared with the conventional IRLS algorithm.

An efficient iterative reweighted least-squares algorithm for two-dimensional FIR filters design in the Lp sense

This paper presents an efficient iterative reweighted least-squares (IRLS) algorithm to obtain an Lp approximation for the design of two-dimensional FIR filters. This algorithm introduces an extra frequency response which implicitly includes the weighting function such that the p-power error to be minimized can be represented in a twodimensional matrix form. The proposed algorithm reduces the computational complexity from O(N6) to O(N3), and storage space from O(N4) to O(N2), compared with the conventional IRLS algorithm.

Determining bed boundaries from inversion of EM logging data using general measures of model structure and data misfit

In this paper we propose a new method to locate bed boundaries by carrying out a 1-D nonlinear inversion of electromagnetic (EM) logging data. We first solve for 1-D resistivity structure in which the earth is modeled using layers of constant thickness. This thickness is determined based upon the tool resolution and the desired resolution from the user. We use general measures of data misfit and model structure in the inversion to construct piecewise‐constant models through the iteratively reweighted least‐squares (IRLS) procedure, and we minimize a generic global model objective function subject to data constraints. The general measure includes the traditional [Formula: see text] norm as a special case. The recovered piecewise‐constant models simulate traditional models comprised of a few uniform layers and hence permit easy determination of bed boundaries. The generic model objective function allows the incorporation of prior geological information and provides measures of the closeness to the reference model and the amount of structure in the recovered model. We develop an efficient scheme to incorporate the IRLS procedure into the search for the appropriate regularization parameter at each iteration. The iterations required by the IRLS algorithm are combined with those for the linearized, iterative inversion procedure. The applicability of the algorithm is demonstrated using synthetic and field data examples.

Improved learning algorithms for mixture of experts in multiclass classification

Mixture of experts (ME) is a modular neural network architecture for supervised learning. A double-loop Expectation-Maximization (EM) algorithm has been introduced to the ME architecture for adjusting the parameters and the iteratively reweighted least squares (IRLS) algorithm is used to perform maximization in the inner loop [Jordan, M.I., Jacobs, R.A. (1994). Hierarchical mixture of experts and the EM algorithm, Neural Computation, 6(2), 181–214]. However, it is reported in literature that the IRLS algorithm is of instability and the ME architecture trained by the EM algorithm, where IRLS algorithm is used in the inner loop, often produces the poor performance in multiclass classification. In this paper, the reason of this instability is explored. We find out that due to an implicitly imposed incorrect assumption on parameter independence in multiclass classification, an incomplete Hessian matrix is used in that IRLS algorithm. Based on this finding, we apply the Newton–Raphson method to the inner loop of the EM algorithm in the case of multiclass classification, where the exact Hessian matrix is adopted. To tackle the expensive computation of the Hessian matrix and its inverse, we propose an approximation to the Newton–Raphson algorithm based on a so-called generalized Bernoulli density. The Newton–Raphson algorithm and its approximation have been applied to synthetic data, benchmark, and real-world multiclass classification tasks. For comparison, the IRLS algorithm and a quasi-Newton algorithm called BFGS have also been applied to the same tasks. Simulation results have shown that the use of the proposed learning algorithms avoids the instability problem and makes the ME architecture produce good performance in multiclass classification. In particular, our approximation algorithm leads to fast learning. In addition, the limitation of our approximation algorithm is also empirically investigated in this paper.

Non-linear inversion using general measures of data misfit and model structure

Abstract We investigate the use of general, non-l 2 measures of data misfit and model structure in the solution of the non-linear inverse problem. Of particular interest are robust measures of data misfit, and measures of model structure which enable piecewise-constant models to be constructed. General measures can be incorporated into traditional linearized, iterative solutions to the non-linear problem through the use of an iteratively reweighted least-squares (IRLS) algorithm. We show how such an algorithm can be used to solve the linear inverse problem when general measures of misfit and structure are considered. The magnetic stripe example of Parker (1994) is used as an illustration. This example also emphasizes the benefits of using a robust measure of misfit when outliers are present in the data. We then show how the IRLS algorithm can be used within a linearized, iterative solution to the non-linear problem. The relevant procedure contains two iterative loops which can be combined in a number of ways. We present two possibilities. The first involves a line search to determine the most appropriate value of the trade-off parameter and the complete solution, via the IRLS algorithm, of the linearized inverse problem for each value of the trade-off parameter. In the second approach, a schedule of prescribed values for the trade-off parameter is used and the iterations required by the IRLS algorithm are combined with those for the linearized, iterative inversion procedure. These two variations are then applied to the 1-D inversion of both synthetic and field time-domain electromagnetic data.