Global Convergence Theorem Research Articles

Modified Newton–SHSS method for a class of systems of nonlinear equations

It has been proved that Newton–HSS method is efficient and robust for solving large sparse systems of nonlinear equations with positive definite Jacobian matrices at the solution points. In this paper, by utilizing the single-step Hermitian and skew-Hermitian splitting (SHSS) iteration technique, which performs efficiently under certain conditions, as the inner solver of the modified Newton method, we propose a class of modified Newton–SHSS methods. Subsequently, the local and semilocal convergence properties of our method will be discussed under some reasonable assumptions. Furthermore, we introduce the modified Newton–SHSS method with a backtracking strategy and analyze its basic global convergence theorem. Finally, several typical instances are used to illustrate the advantages of our methods when the Hermitian part of the Jacobian matrices are dominant.

A penalty method with trust-region mechanism for nonlinear bilevel optimization problem

We present a penalty method with trust-region technique for nonlinear bilevel optimization problem in this paper. This method follows Dennis, El-Alem, and Williamson active set idea and penalty method to transform the nonlinear bilevel optimization problem to unconstrained optimization problem. This method maybe simpler than similar ideas and it does not need to compute a base of the null space. A trust-region technique is used to globalize the algorithm. Global convergence theorem is presented and applications to mathematical programs with equilibrium constraints are given.

A new alternating projection-based prediction–correction method for structured variational inequalities

Recently, some proximal-based alternating direction methods and alternating projection-based prediction–correction methods were proposed to solve the structured variational inequalities in Euclidean space . We note that the proximal-based alternating direction methods need to solve its subproblems exactly. However, the subproblems of the proximal-based alternating direction methods are too difficult to be solved exactly in many practical applications. We also note that the existing alternating projection based prediction–correction methods just can cope with the case that the underlying mappings are Lipschitz continuous. However, it could be difficult to verify their Lipschitz continuity condition, provided that the available information is only the mapping values. In this paper, we present a new alternating projection-based prediction–correction method for solving the structured variational inequalities, where the underlying mappings are continuous. In each iteration, we first employ a new Armijo linesearch to derive the predictors, and then update the next iterate via some minor computations. Under some mild assumptions, we establish the global convergence theorem of the proposed method. Preliminary numerical results are also reported to illustrate the effectiveness of the proposed method.

An interior point method for nonlinear optimization with a quasi-tangential subproblem

We present an interior point method for nonlinear programming in this paper. This method follows Byrd and Omojokun’s idea of step decomposition, which splits the trial step into a normal step and a tangential step. The method employs a new idea of quasi-tangential subproblem, which is used to generate a tangential step that does not lie strictly on the tangent space of the constraints. Quasi-tangential subproblem is finally formulated into an unconstrained quadratic problem by penalizing the constraints. This method is different and maybe simpler than similar ideas, for example, the relaxed tangential step in trust funnel methods (Gould and Toint, 2010; Curtis, et al., 2017). Also, our method does not need to compute a base of the null space. A line search trust-funnel-like strategy is used to globalize the algorithm. Global convergence theorem is presented and applications to mathematical programs with equilibrium constraints are given.

A note on the global convergence theorem of accelerated adaptive Perry conjugate gradient methods

In Andrei (2017), a class of efficient conjugate gradient algorithms (ACGSSV) is proposed for solving large-scale unconstrained optimization problems. However, due to a wrong inequality and an incorrect reasoning used in analyzing the global convergence property for the proposed algorithm, the proof of Theorem 4.2, the global convergence theorem, is incorrect. In this paper, the necessary corrections are made. Under common assumptions, it is shown that Algorithm ACGSSV converges linearly to the unique minimizer.

Comments on “A three-term derivative-free projection method for nonlinear monotone system of equations”

In this note, we show that the proof of Remark 3 of Lemma 3.2 in “A three-term derivative-free projection method for nonlinear monotone system of equations” (Calcolo 53:427–450, 2016) is not correct, which implies that the conclusion of Remark 3 is not appropriate to prove Theorem 3.1. A new proof of Remark 3 is established, which guarantees the corresponding global convergence Theorem 3.1. Throughout, we use the same notations and equation numbers as in the above reference.

Maximum-margin sparse coding

This work devises a maximum-margin sparse coding algorithm, jointly considering reconstruction loss and hinge loss in the model. The sparse representation along with maximum-margin constraint is analogous to kernel trick and maximum-margin properties of support vector machine (SVM), giving a base for the proposed algorithm to perform well in classification tasks. The key idea behind the proposed method is to use labeled and unlabeled data to learn discriminative representations and model parameters simultaneously, making it easier to classify data in the new space. We propose to use block coordinate descent to learn all the components of the proposed model and give detailed derivation for the update rules of the model variables. Theoretical analysis on the convergence of the proposed MMSC algorithm is provided based on Zangwill’s global convergence theorem. Additionally, most previous research studies on dictionary learning suggest to use an overcomplete dictionary to improve classification performance, but it is computationally intensive when the dimension of the input data is huge. We conduct experiments on several real data sets, including Extended YaleB, AR face, and Caltech101 data sets. The experimental results indicate that the proposed algorithm outperforms other comparison algorithms without an overcomplete dictionary, providing flexibility to deal with high-dimensional data sets.

Locality-constrained max-margin sparse coding

This work devises a locality-constrained max-margin sparse coding (LC-MMSC) framework, which jointly considers reconstruction loss and hinge loss simultaneously. Traditional sparse coding algorithms use ℓ1 constraint to force the representation to be sparse, leading to computational expensive process to optimize the objective function. This work uses locality constraint in the framework to preserve information of data locality and avoid the optimization of ℓ1. The obtained representation can achieve the goal of data locality and sparsity. Additionally, this work optimizes coefficients, dictionaries and classification parameters simultaneously, and uses block coordinate descent to learn all the components of the proposed model. This work uses semi-supervised learning approach in the proposed framework, and the goal is to use both labeled data and unlabeled data to achieve accurate classification performance and improve the generalization of the model. We provide theoretical analysis on the convergence of the proposed LC-MMSC algorithm based on Zangwill’s global convergence theorem. This work conducts experiments on three real datasets, including Extended YaleB dataset, AR face dataset and Caltech101 dataset. The experimental results indicate that the proposed algorithm outperforms other comparison algorithms.

Mean Curvature Flow of Mean Convex Hypersurfaces

In the last 15 years, White and Huisken‐Sinestrari developed a far‐reaching structure theory for the mean curvature flow of mean convex hypersurfaces. Their papers provide a package of estimates and structural results that yield a precise description of singularities and of high‐curvature regions in a mean convex flow.In the present paper, we give a new treatment of the theory of mean convex (and k‐convex) flows. This includes: (1) an estimate for derivatives of curvatures, (2) a convexity estimate, (3) a cylindrical estimate, (4) a global convergence theorem, (5) a structure theorem for ancient solutions, and (6) a partial regularity theorem.Our new proofs are both more elementary and substantially shorter than the original arguments. Our estimates are local and universal. A key ingredient in our new approach is the new noncollapsing result of Andrews [2]. Some parts are also inspired by the work of Perelman [32,33]. In a forthcoming paper [17], we will give a new construction of mean curvature flow with surgery based on the methods established in the present paper.Note added in May 2015. Since the first version of this paper was posted on arxiv in April 2013, the estimates have been used to construct mean convex flow with surgery in ℝ3 by Brendle and Huisken [5] in September 2013 and in another paper by the authors in April 2014.© 2016 Wiley Periodicals, Inc.

Comments on a new class of nonlinear conjugate gradient coefficients with global convergence properties

In Rivaie et al. (2012) [1], an efficient CG algorithm has been proposed for solving unconstrained optimization problems. However, due to a wrong inequality (3.3) used in Rivaie et al., the proof of Theorem 2 and the global convergence Theorem 3 are not correct. We present the necessary corrections, then the proposed method in Rivaie et al. still converges globally. Finally, we report some numerical comparisons.

Global convergence of the restarted Lanczos and Jacobi–Davidson methods for symmetric eigenvalue problems

The Lanczos method is well known for computing the extremal eigenvalues of symmetric matrices. For efficiency and robustness a restart strategy is employed in practice, but this makes an analysis of convergence less straightforward. We prove global convergence of the restarted Lanczos method in exact arithmetic using certain convergence properties of the Rayleigh---Ritz procedure, which can be obtained from the discussion by Crouzeix, Philippe, and Sadkane. For the restarted Lanczos method, Sorensen's previous analysis establishes global convergence to the largest eigenvalues under the technical assumption that the absolute values of the off-diagonal elements of the Lanczos tridiagonal matrix are larger than a positive constant throughout the iterations. In this paper, we prove global convergence without any such assumption. The only assumption is that the initial vector is not orthogonal to any of the target exact eigenvectors. More importantly, our results are applicable to dynamic restarting procedures where the dimensions of the projection subspace are dynamically determined. In other words, our analysis can be applied to the recently proposed efficient restart strategies employed in the thick restarted Lanczos method. The convergence theorem is extended to the restarted Lanczos method for computing both the largest and smallest eigenvalues. Moreover, we derive certain global convergence theorems of the block Lanczos and Jacobi---Davidson methods, where, for both algorithms, the Ritz values are shown to converge to exact eigenvalues, although they are not necessarily extremal.

Global convergence of modified multiplicative updates for nonnegative matrix factorization

Nonnegative matrix factorization (NMF) is the problem of approximating a given nonnegative matrix by the product of two nonnegative matrices. The multiplicative updates proposed by Lee and Seung are widely used as efficient computational methods for NMF. However, the global convergence of these updates is not formally guaranteed because they are not defined for all pairs of nonnegative matrices. In this paper, we consider slightly modified versions of the original multiplicative updates and study their global convergence properties. The only difference between the modified updates and the original ones is that the former do not allow variables to take values less than a user-specified positive constant. Using Zangwill's global convergence theorem, we prove that any sequence of solutions generated by either of those modified updates has at least one convergent subsequence and the limit of any convergent subsequence is a stationary point of the corresponding optimization problem. Furthermore, we propose algorithms based on the modified updates that always stop within a finite number of iterations.

On the convergence of a smoothed penalty algorithm for semi-infinite programming

For semi-infinite programming (SIP), we consider a class of smoothed penalty functions, which approximate the exact $$l_\rho (0<\rho \le 1)$$ penalty functions. On base of the smoothed penalty function, we present a feasible penalty algorithm for solving SIP. Without any boundedness condition or coercive condition, we establish the global convergence theorem. Then we present a perturbation theorem for this algorithm and obtain a necessary and sufficient condition for the convergence to the optimal value of SIP. Under Mangasarian–Fromovitz constrained qualification condition, we further discuss the convergence properties for the algorithm based upon a subclass of smooth approximations to the exact $$l_\rho $$ penalty function. Finally, numerical results are given.

Erratum to: Scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization

In Andrei [Scaled memoryless BFGS preconditioned conjugate gradient (CG) algorithm for unconstrained optimization, Optim. Meth. Softw. 22(4) (2007), pp. 561–571], an efficient CG algorithm has been proposed for solving unconstrained optimization problems. However, due to a wrong inequality used in Andrei to show the sufficient descent property for the search directions, the proof of Theorem 2, the global convergence theorem, is incorrect. In what follows, the necessary corrections will be mentioned. Throughout, we use the same notations as in Andrei.

Convergence analysis of modified Newton-HSS method for solving systems of nonlinear equations

Hermitian and skew-Hermitian splitting(HSS) method has been proved quite successfully in solving large sparse non-Hermitian positive definite systems of linear equations. Recently, by making use of HSS method as inner iteration, Newton-HSS method for solving the systems of nonlinear equations with non-Hermitian positive definite Jacobian matrices has been proposed by Bai and Guo. It has shown that the Newton-HSS method outperforms the Newton-USOR and the Newton-GMRES iteration methods. In this paper, a class of modified Newton-HSS methods for solving large systems of nonlinear equations is discussed. In our method, the modified Newton method with R-order of convergence three at least is used to solve the nonlinear equations, and the HSS method is applied to approximately solve the Newton equations. For this class of inexact Newton methods, local and semilocal convergence theorems are proved under suitable conditions. Moreover, a globally convergent modified Newton-HSS method is introduced and a basic global convergence theorem is proved. Numerical results are given to confirm the effectiveness of our method.

Smoothing SQP algorithm for semismooth equations with box constraints

In this paper, in order to solve semismooth equations with box constraints, we present a class of smoothing SQP algorithms using the regularized-smooth techniques. The main difference of our algorithm from some related literature is that the correspondent objective function arising from the equation system is not required to be continuously differentiable. Under the appropriate conditions, we prove the global convergence theorem, in other words, any accumulation point of the iteration point sequence generated by the proposed algorithm is a KKT point of the corresponding optimization problem with box constraints. Particularly, if an accumulation point of the iteration sequence is a vertex of box constraints and additionally, its corresponding KKT multipliers satisfy strictly complementary conditions, the gradient projection of the iteration sequence finitely terminates at this vertex. Furthermore, under local error bound conditions which are weaker than BD-regular conditions, we show that the proposed algorithm converges superlinearly. Finally, the promising numerical results demonstrate that the proposed smoothing SQP algorithm is an effective method.

Comments on: “A family of derivative-free conjugate gradient methods for large-scale nonlinear systems of equations” [J. Comput. Appl. Math. 224 (2009) 11–19

In this note a simple counter example shows that the proof of Lemma 3.3 in [1, W. Cheng, Y. Xiao and Q. Hu, A family of derivative-free conjugate gradient methods for large-scale nonlinear systems of equations, J. Comput. Appl. Math. 224 (2009) 11–19] is not correct, which implies that Lemma 3.2 in [1] is not enough to ensure Lemma 3.3 in [1]. A new proof is given, which leads to a stronger result than Lemma 3.2 in [1]. And this result not only guarantees that Lemma 3.3 in [1] holds, but also improves the corresponding global convergence Theorem 3.1 in [1].

Approximate global convergence and quasireversibility for a coefficient inverse problem with backscattering data

A numerical method possessing the approximate global convergence property is developed for a 3-D coefficient inverse problem for hyperbolic partial differential equations with backscattering data resulting from a single measurement. An important part of this technique is the quasireversibility method. An approximate global convergence theorem is proved. Results of two numerical experiments are presented.

A note on the global convergence theorem of the scaled conjugate gradient algorithms proposed by Andrei

In (Andrei, Comput. Optim. Appl. 38:402---416, 2007), the efficient scaled conjugate gradient algorithm SCALCG is proposed for solving unconstrained optimization problems. However, due to a wrong inequality used in (Andrei, Comput. Optim. Appl. 38:402---416, 2007) to show the sufficient descent property for the search directions of SCALCG, the proof of Theorem 2, the global convergence theorem of SCALCG, is incorrect. Here, in order to complete the proof of Theorem 2 in (Andrei, Comput. Optim. Appl. 38:402---416, 2007), we show that the search directions of SCALCG satisfy the sufficient descent condition. It is remarkable that the convergence analyses in (Andrei, Optim. Methods Softw. 22:561---571, 2007; Eur. J. Oper. Res. 204:410---420, 2010) should be revised similarly.

A Wilkinson-like multishift QR algorithm for symmetric eigenvalue problems and its global convergence

In 1989, Bai and Demmel proposed the multishift QR algorithm for eigenvalue problems. Although the global convergence property of the algorithm (i.e., the convergence from any initial matrix) still remains an open question for general nonsymmetric matrices, in 1992 Jiang focused on symmetric tridiagonal case and gave a global convergence proof for the generalized Rayleigh quotient shifts. In this paper, we propose Wilkinson-like shifts, which reduce to the standard Wilkinson shift in the single shift case, and show a global convergence theorem.