Global Convergence Results Research Articles

Error analysis of a finite element method with GMMP temporal discretisation for a time-fractional diffusion equation

A time-fractional initial–boundary value problem is considered, where the spatial domain has dimension d∈{1,2,3}, the spatial differential operator is a standard elliptic operator, and the time derivative is a Caputo derivative of order α∈(0,1). To discretise in space we use a standard piecewise-polynomial finite element method, while for the temporal discretisation the GMMP scheme (a variant of the Grünwald-Letnikov scheme) is used on a uniform mesh. The analysis of the GMMP scheme for solutions that exhibit a typical weak singularity at the initial time t=0 has not previously been considered in the literature. A global convergence result is derived in L∞(L2), then a more delicate analysis of the error in this norm shows that, away from t=0, the method attains optimal-rate convergence. Numerical results confirm the sharpness of the theoretical error bounds.

A new accelerated conjugate gradient method for large-scale unconstrained optimization

In this paper, we present a new conjugate gradient method using an acceleration scheme for solving large-scale unconstrained optimization. The generated search direction satisfies both the sufficient descent condition and the Dai–Liao conjugacy condition independent of line search. Moreover, the value of the parameter contains more useful information without adding more computational cost and storage requirements, which can improve the numerical performance. Under proper assumptions, the global convergence result of the proposed method with a Wolfe line search is established. Numerical experiments show that the given method is competitive for unconstrained optimization problems, with a maximum dimension of 100,000.

Large-scale quasi-Newton trust-region methods with low-dimensional linear equality constraints

We propose two limited-memory BFGS (L-BFGS) trust-region methods for large-scale optimization with linear equality constraints. The methods are intended for problems where the number of equality constraints is small. By exploiting the structure of the quasi-Newton compact representation, both proposed methods solve the trust-region subproblems nearly exactly, even for large problems. We derive theoretical global convergence results of the proposed algorithms, and compare their numerical effectiveness and performance on a variety of large-scale problems.

A family of Hager–Zhang conjugate gradient methods for system of monotone nonlinear equations

This paper presents two modified Hager–Zhang (HZ) Conjugate Gradient methods for solving large-scale system of monotone nonlinear equations. The methods were developed by combining modified forms of the one-parameter method by Hager and Zhang (2006) and the hyperplane projection technique. Global convergence and numerical results of the methods are established. Preliminary numerical results show that the proposed methods are promising and more efficient compared to the methods presented by Mushtak and Keyvan (2018) and Sun et al. (2017).

An Alternating Augmented Lagrangian method for constrained nonconvex optimization

ABSTRACTWe consider the problem of minimizing a smooth nonconvex function over a structured convex feasible set, that is, defined by two sets of constraints that are easy to treat when considered separately. In order to exploit the structure of the problem, we define an equivalent formulation by duplicating the variables and we consider the augmented Lagrangian of this latter formulation. Following the idea of the Alternating Direction Method of Multipliers (ADMM), we propose an algorithm where a two-blocks decomposition method is embedded within an augmented Lagrangian framework. The peculiarities of the proposed algorithm are the following: (1) the computation of the exact solution of a possibly nonconvex subproblem is not required; (2) the penalty parameter is iteratively updated once an approximated stationary point of the augmented Lagrangian is determined. Global convergence results are stated under mild assumptions and without requiring convexity of the objective function. Although the primary aim of the paper is theoretical, we perform numerical experiments on a nonconvex problem arising in machine learning, and the obtained results show the practical advantages of the proposed approach with respect to classical ADMM.

A Flexible Framework for Cubic Regularization Algorithms for Nonconvex Optimization in Function Space

We propose a cubic regularization algorithm that is constructed to deal with nonconvex minimization problems in function space. It allows for a flexible choice of the regularization term and thus accounts for the fact that in such problems one often has to deal with more than one norm. Global and local convergence results are established in a general framework.

Particle swarm optimization with damping factor and cooperative mechanism

A novel variant of particle swarm optimization with damping factor and cooperation mechanism (PSO-DFCM) to search the global optima in a large scale and high-dimensional searching space. In this optimal searching strategy, one balances the exploring and exploiting abilities of particles by introducing a new parameter, a damping factor α, which is used to adjust the position information inherited from the previous state. The cooperative mechanism between the global-best-oriented and the local-best-oriented swarms is employed to help find the global optima quickly. In order to reduce the negative effect of unfavorable particles on swarm evolution, a new concept of evolution history, the least optimal particle in individuals’ histories — pleast, is defined to decide whether current information of particles is abandoned and reinitialized in our proposal. Also, fuzzy c-means clustering is applied to cluster the particles’ positions for the neighborhood establishment of individuals. Our comparative study on benchmark test functions demonstrates that the proposed PSO outperforms the standard PSO and three state-of-art variants of PSO in terms of global optimum convergence and final optimal results.

A new class of conjugate gradient methods for unconstrained smooth optimization and absolute value equations

In this paper, we introduce a new three-term conjugate gradient (NTTCG) method to solve unconstrained smooth optimization problems. NTTCG is based on conjugate gradient methods proposed by Dai and Yuan (SIAM J Optim 10:177–182, 1999) and Polak and Ribiere (Rev Francaise Inform Rech Oper 3(16):35–43, 1969). The descent property of the direction generated by NTTCG in each iteration is established. Under some standard assumptions, the global convergence results of the new methods are investigated. The extension of this algorithm is proposed to solve absolute value equations (AVE), called three-term conjugate subgradient (NTTCS) method. Numerical experiments are reported for unconstrained CUTEst problems and AVE.

A New Proximal Iterative Hard Thresholding Method with Extrapolation for $$\ell _0$$ ℓ 0 Minimization

In this paper, we consider a non-convex problem which is the sum of $$\ell _0$$ -norm and a convex smooth function under a box constraint. We propose one proximal iterative hard thresholding type method with an extrapolation step for acceleration and establish its global convergence results. In detail, the sequence generated by the proposed method globally converges to a local minimizer of the objective function. Finally, we conduct numerical experiments to show the proposed method’s effectiveness on comparison with some other efficient methods.

A three term Polak-Ribière-Polyak conjugate gradient method close to the memoryless BFGS quasi-Newton method

<p style='text-indent:20px;'>In this paper, we develop a three-term Polak-Ribière-Polyak conjugate gradient method, in which the search direction is close to the direction in the memoryless BFGS quasi-Newton method. The new scheme reduces to the standard Polak-Ribière-Polyak method when an exact line search is used. For any line search, the method satisfies the sufficient descent condition <inline-formula><tex-math id="M1">\begin{document}$g_{k}^{T}d_{k}≤ -{(1-\frac{1}{4}(1+\overline{t})^2})\|g_k\|^2$\end{document}</tex-math></inline-formula>, where <inline-formula><tex-math id="M2">\begin{document}$\overline{t}∈[0,1)$\end{document}</tex-math></inline-formula> is a constant. The global convergence results of the new algorithm are established with suitable line search methods. Numerical results show that the proposed method is efficient for the unconstrained problems in the CUTEr library.

Performance analysis of two EM-based measurement bias estimation processes for tracking systems

In target tracking, the measurements collected by sensors can be biased in some real scenarios, e.g., due to systematic error. To accurately estimate the target trajectory, it is essential that the measurement bias be identified in the first place. We investigate the iterative bias estimation process based on the expectation-maximization (EM) algorithm, for cases where sufficiently large numbers of measurements are at hand. With the assistance of extended Kalman filtering and smoothing, we derive two EM estimation processes to estimate the measurement bias which is formulated as a random variable in one state-space model and a constant value in another. More importantly, we theoretically derive the global convergence result of the EM-based measurement bias estimation and reveal the link between the two proposed EM estimation processes in the respective state-space models. It is found that the bias estimate in the second state-space model is more accurate and of less complexity. Furthermore, the EM-based iterative estimation converges faster in the second state-space model than in the first one. As a byproduct, the target trajectory can be simultaneously estimated with the measurement bias, after processing a batch of measurements. These results are confirmed by our simulations.

Solving a large scale nonlinear unconstrained optimization problems by using new coefficient of conjugate gradient method with exact line search direction

In this paper, an efficient modification of nonlinear conjugate gradient method and an associated implementation, based on an exact line search, are proposed and analyzed to solve large-scale unconstrained optimization problems. The method satisfies the sufficient descent property. Furthermore, global convergence result is proved. Computational results for a set of unconstrained optimization test problems, some of them from CUTE library, showed that this new conjugate gradient algorithm seems to converge more stable and outperforms the other similar methods in many situations.

The numerical solution of the semi-explicit IDAEs by discontinuous piecewise polynomial approximation

In this paper, we consider a semi-explicit form of Volterra integro-differential-algebraic equations (IDAEs) and investigate the existence and uniqueness of solution of these systems by using differentiability index. Numerical method based on discontinuous piecewise polynomial approximation is proposed for the solution of the semi-explicit IDAEs and global convergence results are established. The performance of the numerical scheme is illustrated by means of some test problems.

Two classes of Ljusternik–Schnirelman minimax algorithms and an application for finding multiple negative energy solutions of a class of p-Laplacian equations

In this paper, two classes of LS-minimax algorithms are presented, they are applied to numerically find multiple negative energy solutions of the p-Laplacian equation −Δpu=λ|u|r−1u+|u|q−1u,x∈Ω⊂Rl,u=0,x∈∂Ω,where Ω is an open bounded domain, 0<r<p−1<q<p∗−1, λ>0, and p∗ is the Sobolev exponent, and mathematical justification and global convergence result for them are established. By combining LS-minimax algorithm with the finite element method, it is verified that, as element size goes to zero, numerical solutions of p-Laplacian equation captured by LS-minimax algorithm converge to solutions of p-Laplacian equation. Two LS-minimax algorithms developed in [1] are two special algorithms in these two classes of algorithms.

Extended Dai-Yuan conjugate gradient strategy for large-scale unconstrained optimization with applications to compressive sensing

We present a new spectral conjugate gradient method based on the Dai-Yuan strategy to solve large-scale unconstrained optimization problems with applications to compressive sensing. In our method, the numerator of conjugate gradient parameter is a convex combination from the maximum gradient norm value in some preceding iterates and the current gradient norm value. This combination will try to produce the larger step-size far away from the optimizer and the smaller step-size close to it. In addition, the spectral parameter guarantees the descent property of the new generated direction in each iterate. The global convergence results are established under some standard assumptions. Numerical results are reported which indicate the promising behavior of the new procedure to solve large-scale unconstrained optimization and compressive sensing problems.

A new class of nonlinear conjugate gradient with global convergence properties

Nonlinear conjugate gradient methods are widely used in solving large scale unconstrained optimization. Their wide application in many fields are due to their low memory. Numerous studies have been conducted recently to improve these methods. In this paper, a new class of conjugate gradient coefficients that possess global convergence properties is proposed. The global convergence result using exact line searches are discussed. Numerical result shows that the proposed method is more efficient compared with the other classical conjugate gradient.

Material Optimization in Transverse Electromagnetic Scattering Applications

A class of algorithms for the solution of discrete material optimization problems in electromagnetic applications is discussed. The idea behind the algorithm is similar to that of the sequential programming. However, in each major iteration a model is established on the basis of an appropriately parametrized material tensor. The resulting nonlinear parametrization is treated on the level of the subproblem, for which globally optimal solutions can be computed due to the block separability of the model. Although global optimization of nonconvex design problems is generally prohibitive, a well chosen combination of analytic solutions along with standard global optimization techniques leads to a very efficient algorithm for most relevant material parametrizations. A global convergence result for the overall algorithm is established. The effectiveness of the approach in terms of both computation time and solution quality is demonstrated by numerical examples based on a two-dimensional Helmholtz equation, inclu...

A New Conjugate Gradient Method with Smoothing $$L_{1/2} $$ L 1 / 2 Regularization Based on a Modified Secant Equation for Training Neural Networks

Proposed in this paper is a new conjugate gradient method with smoothing $$L_{1/2} $$ regularization based on a modified secant equation for training neural networks, where a descent search direction is generated by selecting an adaptive learning rate based on the strong Wolfe conditions. Two adaptive parameters are introduced such that the new training method possesses both quasi-Newton property and sufficient descent property. As shown in the numerical experiments for five benchmark classification problems from UCI repository, compared with the other conjugate gradient training algorithms, the new training algorithm has roughly the same or even better learning capacity, but significantly better generalization capacity and network sparsity. Under mild assumptions, a global convergence result of the proposed training method is also proved.

An alternative globalization strategy for unconstrained optimization

We propose a new globalization strategy that can be used in unconstrained optimization algorithms to support rapid convergence from remote starting points. Our approach is based on using multiple points at each iteration to build a sequence of representative models of the objective function. Using the new information gathered from those multiple points, a local step is gradually improved by updating its direction as well as its length. We give a global convergence result and also provide the parallel implementation details accompanied with a numerical study. Our numerical study shows that the proposed algorithm is a promising alternative as a globalization strategy.

An implicit filtering algorithm for derivative-free multiobjective optimization with box constraints

This paper is concerned with the definition of new derivative-free methods for box constrained multiobjective optimization. The method that we propose is a non-trivial extension of the well-known implicit filtering algorithm to the multiobjective case. Global convergence results are stated under smooth assumptions on the objective functions. We also show how the proposed method can be used as a tool to enhance the performance of the Direct MultiSearch (DMS) algorithm. Numerical results on a set of test problems show the efficiency of the implicit filtering algorithm when used to find a single Pareto solution of the problem. Furthermore, we also show through numerical experience that the proposed algorithm improves the performance of DMS alone when used to reconstruct the entire Pareto front.