Methods For Least Squares Problems Research Articles

An efficient real representation method for least squares problem of the quaternion constrained matrix equation AXB + CY D = E

Let and represent the sets of all η-Hermitian quaternion matrices and η-anti-Hermitian quaternion matrices, respectively. On the basis of the real representation matrix of a quaternion matrix and its particular structure, we convert the least squares problem of the quaternion matrix equation AXB + CY D = E over into the corresponding problem of the real matrix equation over free variables, and then we establish its unique minimal norm least squares solution. Our resulting expressions are expressed only by real matrices, and the algorithm only includes real operations. Consequently, they are very simple and convenient. Compared with the existing method [S.F. Yuan, Q.W. Wang, and X. Zhang, Least-squares problem for the quaternion matrix equation AXB + CYD = E over different constrained matrices, Int. J. Comput. Math. 90 (2013), pp. 565–576], the final two examples show that our method is more efficient and superior.

A novel iterative integration regularization method for ill-posed inverse problems

This paper proposes a new iterative integration regularization method for robust solution of ill-posed inverse problems. The proposed method is motivated from the fact that inversion of a positive definite matrix can be expressed in an integral form. Then, the development of the proposed method is mainly twofold. Firstly, two ways—including the linear iteration and the exponential ($$2^j$$) iteration—are invoked to compute the integral, of which the exponential iteration is often preferred due to its fast convergence. Secondly, after stability analysis, the proposed method is shown able to filter out the undesired effect of relatively small singular values, while preserving the desired terms of relatively large singular values, i.e., the proposed method has the guaranteed regularization effect. Numerical examples on three typical ill-posed problems are conducted with detailed comparison to some usual direct and iterative regularization methods. Final results have highlighted the proposed method: (a) due to the iterative nature, the proposed method often turns out to be more efficient than the conventional direct regularization methods including the Tikhonov regularization and the truncated singular value decomposition (TSVD), (b) the proposed method converges much faster than the Landweber method and (c) the regularization effect is guaranteed in the proposed method, while may not be in the conjugate gradient method for least squares problem (CGLS).

Iterative Methods for Least Squares Problem in Split Quaternionic Mechanics

Iterative Methods for Least Squares Problem in Split Quaternionic Mechanics

KRYLOV SUBSPACE METHODS WITH DEFLATION AND BALANCING PRECONDITIONERS FOR LEAST SQUARES PROBLEMS

For solving least squares problems, the CGLS method is a typical method in the point of view of iterative methods. When the least squares problems are ill-conditioned, the convergence behavior of the CGLS method will present a deteriorated result. We expect to select other iterative Krylov subspace methods to overcome the disadvantage of CGLS. Here the GMRES method is a suitable algorithm for the reason that it is derived from the minimal residual norm approach, which coincides with least squares problems. Ken Hayami proposed BAGMRES for solving least squares problems in [GMRES Methods for Least Squares Problems, SIAM J. Matrix Anal. Appl., 31(2010), pp.2400-2430]. The deflation and balancing preconditioners can optimize the convergence rate through modulating spectral distribution. Hence, in this paper we utilize preconditioned iterative Krylov subspace methods with deflation and balancing preconditioners in order to solve ill-conditioned least squares problems. Numerical experiments show that the methods proposed in this paper are better than the CGLS method.

Convergence analysis of Riemannian Gauss–Newton methods and its connection with the geometric condition number

We obtain estimates of the multiplicative constants appearing in local convergence results of the Riemannian Gauss-Newton method for least squares problems on manifolds and relate them to the geometric condition number of [P. B\"urgisser and F. Cucker, Condition: The Geometry of Numerical Algorithms, 2013].

Algebraic methods for least squares problem in split quaternionic mechanics

A split quaternionic least squares problem is one method of solving overdetermined sets of split quaternion linear equations AX ≈ B that is appropriate when there is error in the matrix B. In this paper, by means of complex representation and real representation of a split quaternion matrix, we study the split quaternionic least squares problem, derive two algebraic methods for finding solutions of the problems in split quaternionic mechanics.

Optimization-based features extraction for K-complex detection

The K-complex is a transient electroencephalogram (EEG, brain activity) waveform that contributes to sleep stage scoring. An automated detection of K-complexes is an important component of sleep stage monitoring. This automation is difficult due to the stochastic nature of brain signals, presence of noise, complexity, and extreme size of data. We develop an optimization model, based on solving a sequence of linear least squares problems, to extract key features of EEG signals. The proposed approach significantly reduces the dimension of the problem and the computational time while the classification accuracy is enhanced in most cases. Numerical results show that this procedure is efficient in detecting K-complexes. References R. Agarwal and J. Gotman. Digital tools in polysomnography. J. Clin. Neurophysiol., 19(2):136–143, 2002. http://journals.lww.com/clinicalneurophys/Abstract/2002/03000/Digital_Tools_in_Polysomnography.4.aspx . J. L. Barlow. Numerical aspects of solving linear least squares problems. Technical report, Computer Science Department, The Pennsylvania State University, University Park, PA, USA, January 1999. www.cse.psu.edu/ barlow/book.ps . A. Bjorck. Numerical Methods for Least Squares Problems. Handbook of Numerical Analysis. SIAM, 1996. doi:10.1137/1.9781611971484 . S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, New York, NY, USA, 2010. http://www.cambridge.org/au/academic/subjects/statistics-probability/optimization-or-and-risk/convex-optimization?format=HB . G. Bremer, J. R. Smith, and I. Karacan. Automatic detection of the K-complex in sleep electroencephalograms. IEEE T. Bio-Med. Eng., 17(4):314–323, 1970. doi:10.1109/TBME.1970.4502759 . P. R. Carney, R. B. Berry, and J. D. Geyer. Clinical Sleep Disorders. LWW medical book collection. Lippincott Williams and Wilkins, 2005. http://www.lww.com/Product/9780781786928 . G. H. Golub and C. F. Van Loan. Matrix Computations. Johns Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, Baltimore, MD, USA, 1996. http://portal.acm.org/citation.cfm?id=248979 . C. Iber, S. Ancoli-Israel, A. L. Chesson, and S. F. Quani. AASM manual for the scoring of sleep and associated events. Rules, technology and technical specifications. Technical report, AASM, Westchester, IL, 2007. http://www.aasmnet.org/scoringmanual/v2.0.2/html/index.html?IXDevelopmentProcess.html . B. H. Jansen. Artificial neural nets for K-complex detection. IEEE Eng. Med. Biol., 9(3):50–52, 1990. doi:10.1109/51.59213 . A. Kales, A. Rechtschaffen, Los Angeles University of California, and NINDB Neurological Information Network (U.S.). A manual of standardized terminology, techniques and scoring system for sleep stages of human subjects. U. S. National Institute of Neurological Diseases and Blindness, Neurological Information Network Bethesda, Md, 1968. http://nla.gov.au/nla.cat-vn823711 . J. E. Mitchell. Branch-and-cut algorithms for combinatorial optimization problems. In P. M. Pardalos and M. G. C. Resende, editors, Handbook of Applied Optimization, pages 65–77. Oxford University Press, 2002. http://global.oup.com/academic/product/handbook-of-applied-optimization-9780195125948?cc=au&lang=en& . D. Moloney, N. Sukhorukova, P. Vamplew, J. Ugon, G. Li, G. Beliakov, C. Philippe, H. Amiel, and A. Ugon. Detecting K-complexes for sleep stage identification using nonsmooth optimization. ANZIAM J., 52:319–332, 2011. doi:10.1017/S1446181112000016 . O. Sheriff, B. Pagnrek, S. Mamouhd, and R. Broughton. Automatic detection of K-complex in sleep EEG. Int. Electrical Electronic Conf. Exp., 81, 1977. Z. Tang and N. Ishii. Detection of the K-complex using a new method of recognizing waveform based on the discrete wavelet transform. IEICE T. Inf. Syst., E78–D(1):77–85, 1995. http://search.ieice.org/bin/summary.php?id=e78-d_1_77 . L. N. Trefethen and D. Bau. Numerical Linear Algebra. SIAM, 1997. http://bookstore.siam.org/ot50/ . Weka web site. www.cs.waikato.ac.nz/ml/weka/ .

Inner-Iteration Krylov Subspace Methods for Least Squares Problems

Stationary inner iterations in combination with Krylov subspace methods are proposed for overdetermined least squares problems. The inner iterations are efficient in terms of computational work and memory and also serve as powerful preconditioners for ill-conditioned and rank-deficient problems. Theoretical justifications for using the inner iterations as preconditioners are presented. Numerical experiments on overdetermined sparse least squares problems show that the proposed methods outperform previous methods, especially for ill-conditioned and rank-deficient problems.

GMRES Methods for Least Squares Problems

The standard iterative method for solving large sparse least squares problems $\min\|\mbox{\boldmathb}-A\mbox{\boldmathx}\|_2$, $A\in\mathbf{R}^{m\times n}$, is the CGLS method, or its stabilized version, LSQR, which is mathematically equivalent to applying the conjugate gradient method to the normal equation $A^{\mbox{\tiny T}}A\mbox{\boldmathx}=A^{\mbox{\tiny T}}\mbox{\boldmathb}$. We consider alternative methods using a matrix $B\in\mathbf{R}^{n\times m}$ and applying the generalized minimal residual (GMRES) method to $\min\|\mbox{\boldmathb}-AB\mbox{\boldmathz}\|_2$ or $\min\|B\mbox{\boldmathb}-BA\mbox{\boldmathx}\|_2$. We give a sufficient condition concerning B for the GMRES methods to give a least squares solution without breakdown for arbitrary $\mbox{\boldmathb}$, for overdetermined, underdetermined, and possibly rank-deficient problems. We then give a convergence analysis of the GMRES methods as well as the CGLS method. Then, we propose using the robust incomplete factorization (RIF) for B. Finally, we show by numerical experiments on overdetermined and underdetermined problems that, for ill-conditioned problems, the GMRES methods with RIF give least squares solutions faster than the CGLS and LSQR methods with RIF, and are similar in performance to the reorthogonalized CGLS with RIF.

Preconditioned GMRES methods for least squares problems

For least squares problems of minimizing ∥b −Ax∥2 whereA is a large sparsem ×n (m ≥n) matrix, the common method is to apply the conjugate gradient method to the normal equationA T Ax =A T b. However, the condition number ofA T A is square of that ofA, and convergence becomes problematic for severely ill-conditioned problems even with preconditioning. In this paper, we propose two methods for applying the GMRES method to the least squares problem by using ann ×m matrixB. We give the necessary and sufficient condition thatB should satisfy in order that the proposed methods give a least squares solution. Then, for implementations forB, we propose an incomplete QR decomposition IMGS(l). Numerical experiments showed that the simplest casel = 0 gives the best results, and converges faster than previous methods for severely ill-conditioned problems. The preconditioner IMGS(0) is equivalent to the caseB = (diag(A T A))−1 A T, so (diag(A T A))−1 A T was the best preconditioner among IMGS(l) and Jennings’ IMGS(τ). On the other hand, CG-IMGS(0) was the fastest for well-conditioned problems.

Accelerated generalized successive overrelaxation method for least squares problems

In this paper we develop a new iterative method for solving large and sparse least squares problems. We accelerate the generalized successive overrelaxation method. The advantage of this method is that to cover some of difficulties of the well known methods such as GS, SOR and AOR [D. Young, Iterative Solution of Large Linear Systems, Academic Press, New York, London, 1971 [1]; G. Meurant, Computers Solution of Linear Systems Studies in Mathematics and its Applications, vol. 28, Elsevier/North Holland, 1999 [2]], GSOR [C. Li, B. Li, J. Evans, A generalized successive overrelaxation method for least squares problems, BIT 2(2) (1998) 347–355; C. Li, B. Li, J. Evans, Optimum acceleration parameter for the GSOR method, Neural, Parallel and Scientific Computations 7 (1999) 453–462]. AGSOR method involved two-parameter. By using AGSOR we can solve large linear systems in ill-condition problems. These ill-posed systems has zeroes block or sparse coefficients matrix, even we can not improve ill-condition by interchanging rows and columns.

Parallel MCGLS and ICGLS Methods for Least Squares Problems on Distributed Memory Architectures

In this paper we mainly study the parallelization of the CGLS method, a basic iterative method for large and sparse least squares problems in which the conjugate gradient method is applied to solve normal equations. On modern parallel architectures its parallel performance is always limited because of the global communication required for inner products, the main bottleneck of parallel performance. In this paper, we describe a modified CGLS (MCGLS) method which improve parallel performance by assembling the results of a number of inner products collectively and by creating situations where communication can be overlapped with computation. More importantly, we also propose an improved CGLS (ICGLS) method to reduce inner product's global synchronization points to half, then significantly improve the parallel performance accordingly compared with the standard CGLS method and the MCGLS method.

Accurate conjugate gradient methods for families of shifted systems

We consider the solution of the linear system A T A +σ I x σ= A T b , for various real values of σ. This family of shifted systems arises, for example, in Tikhonov regularization and computations in lattice quantum chromodynamics. For each single shift σ this system can be solved using the conjugate gradient method for least squares problems (CGLS). In literature various implementations of the, so-called, multishift CGLS methods have been proposed. These methods are mathematically equivalent to applying the CGLS method to each shifted system separately but they solve all systems simultaneously and require only two matrix–vector products (one by A and one by A T ) and two inner products per iteration step. Unfortunately, numerical experiments show that, due to roundoff errors, in some cases these implementations of the multishift CGLS method can only attain an accuracy that depends on the square of condition number of the matrix A . In this paper we will argue that, in the multishift CGLS method, the impact on the attainable accuracy of rounding errors in the Lanczos part of the method is independent of the effect of roundoff errors made in the construction of the iterates. By making suitable design choices for both parts, we derive a new (and efficient) implementation that tries to remove the limitation of previous proposals. A partial roundoff error analysis and various numerical experiments show promising results.

Accuracy of preconditioned CG-type methods for least squares problems

The conjugate gradient method with IMGS, an incomplete modified version of Gram-Schmidt orthogonalization to obtain an incomplete orthogonal factorization preconditioner, applied to the normal equations (PCGLS) is often used as the basic iterative method to solve the linear least squares problems. In this paper, a detailed analysis is given for understanding the effect of rounding errors on IMGS and determining the accuracy of computed solutions of PCGLS with IMGS for linear least squares problems in finite precision. It is shown that for a consistent system, the difference between the true residuals and the updated approximate residual vectors generated depends on the machine precision ε, on the maximum growth in norm of the iterates over their initial values, the norm of the true solution, and the condition number of R which is affected by the drop set in incomplete Gram-Schmidt factorization. Similar results are obtained for the difference between the true and computed solution for inconsistent systems. Numerical tests are carried out to confirm the theoretical conclusions.

A Projection Method for Least Squares Problems with a Quadratic Equality Constraint

We consider the least squares problem with a quadratic equality constraint (LSQE), i.e., minimizing |Ax-b|2 subject to $\|x\|_2=\alpha$, without the assumption $\|A^\dagger b\|_2>\alpha$ which is commonly imposed in the literature. Structure and perturbation analysis are given to demonstrate the sensitivity of the LSQE problem. We present a projection method combined with correction techniques (PMCT) for solving numerically the LSQE problem when the LSQE problem is ill-conditioned. We also give a detailed convergence analysis of our algorithms to illustrate the convergence behavior. Our algorithms have some obvious advantages over Newton's method and variants. Numerical experiments indicate that PMCT is much more efficient than Newton's methods when the LSQE problem is ill-conditioned; PMCT has a 90% success rate in terms of convergence, while commonly used Newton-type iterations almost always fail.

A generalized successive overrelaxation method for least squares problems

In this paper a new iterative method is given for solving large sparse least squares problems and computing the minimum norm solution to underdetermined consistent linear systems. The new scheme is called the generalized successive overrelaxation (GSOR) method and is shown to be convergent ifA is full column rank. The GSOR method involves a parameter ρ and an auxiliary matrixP. One can choose matrix P so that the GSOR method only involves matrix and vector operations; therefore the GSOR method is suitable for parallel computations. Besides, the GSOR method can be combined with preconditioning techniques, and therefore can be expected to be more effective.

On the weighting method for least squares problems with linear equality constraints

The weighting method for solving a least squares problem with linear equality constraints multiplies the constraints by a large number and appends them to the top of the least squares problem, which is then solved by standard techniques. In this paper we give a new analysis of the method, based on the QR decomposition, that exhibits many features of the algorithm. In particular it suggests a natural criterion for chosing the weighting factor.

A New Class of Incremental Gradient Methods for Least Squares Problems

The least mean squares (LMS) method for linear least squares problems differs from the steepest descent method in that it processes data blocks one-by-one, with intermediate adjustment of the parameter vector under optimization. This mode of operation often leads to faster convergence when far from the eventual limit and to slower (sublinear) convergence when close to the optimal solution. We embed both LMS and steepest descent, as well as other intermediate methods, within a one-parameter class of algorithms, and we propose a hybrid class of methods that combine the faster early convergence rate of LMS with the faster ultimate linear convergence rate of steepest descent. These methods are well suited for neural network training problems with large data sets. Furthermore, these methods allow the effective use of scaling based, for example, on diagonal or other approximations of the Hessian matrix.

Numerical methods for least squares problems

Preface 1. Mathematical and statistical properties of least squares solutions 2. Basic numerical methods 3. Modified least squares problems 4. Generalized least squares problems 5. Constrained least squares problems 6. Direct methods for sparse problems 7. Iterative methods for least squares problems 8. Least squares problems with special bases 9. Nonlinear least squares problems Bibliography Index.

The Block Successive Relaxation Additive Tterative method for least squares problems

In this paper, a Block Successive Relaxation Additive Iterative method is considered for 2 cyclic block matrices derived from least squares problems. The convergence conditions and the optimum block relaxation parameter for the method are derived.