Methods Softw Research Articles

Solving $$\ell _0$$ ℓ 0 -penalized problems with simple constraints via the Frank–Wolfe reduced dimension method

$$\ell _0$$ -penalized problems arise in a number of applications in engineering, machine learning and statistics, and, in the last decades, the design of algorithms for these problems has attracted the interest of many researchers. In this paper, we are concerned with the definition of a first-order method for the solution of $$\ell _0$$ -penalized problems with simple constraints. We use a reduced dimension Frank–Wolfe algorithm Rinaldi (Optim Methods Softw, 26, 2011) and show that the subproblem related to the computation of the Frank–Wolfe direction can be solved analytically at least for some sets of simple constraints. This gives us a very easy to implement and quite general tool for dealing with $$\ell _0$$ -penalized problems. The proposed method is then applied to the numerical solution of two practical optimization problems, namely, the Sparse Principal Component Analysis and the Sparse Reconstruction of Noisy Signals. In both cases, the reported numerical performances and comparisons with state-of-the-art solvers show the efficiency of the proposed method.

Boundedness of a Type of Iterative Sequences in Two-Dimensional Quadratic Programming

We prove that any iterative sequence generated by the projection decomposition algorithm of Pham Dinh et al. (Optim Methods Softw 23:609---629, 2008) in quadratic programming is bounded, provided that the quadratic program in question is two-dimensional and solvable.

Two modified three-term conjugate gradient methods with sufficient descent property

Based on the insight gained from the three-term conjugate gradient methods suggested by Zhang et al. (Optim Methods Softw 22:697–711, 2007) two nonlinear conjugate gradient methods are proposed, making modifications on the conjugate gradient methods proposed by Dai and Liao (Appl Math Optim 43:87–101, 2001), and Zhou and Zhang (Optim Methods Softw 21:707–714, 2006). The methods can be regarded as modified versions of two three-term conjugate gradient methods proposed by Sugiki et al. (J Optim Theory Appl 153:733–757, 2012) in which the search directions are computed using the secant equations in a way to achieve the sufficient descent property. One of the methods is shown to be globally convergent for uniformly convex objective functions while the other is shown to be globally convergent without convexity assumption on the objective function. Comparative numerical results demonstrating efficiency of the proposed methods are reported.

The generalized trust region subproblem

The interval bounded generalized trust region subproblem (GTRS) consists in minimizing a general quadratic objective, q 0(x)?min, subject to an upper and lower bounded general quadratic constraint, l≤q 1(x)≤u. This means that there are no definiteness assumptions on either quadratic function. We first study characterizations of optimality for this implicitly convex problem under a constraint qualification and show that it can be assumed without loss of generality. We next classify the GTRS into easy case and hard case instances, and demonstrate that the upper and lower bounded general problem can be reduced to an equivalent equality constrained problem after identifying suitable generalized eigenvalues and possibly solving a sparse system. We then discuss how the Rendl-Wolkowicz algorithm proposed in Fortin and Wolkowicz (Optim. Methods Softw. 19(1):41---67, 2004) and Rendl and Wolkowicz (Math. Program. 77(2, Ser. B):273---299, 1997) can be extended to solve the resulting equality constrained problem, highlighting the connection between the GTRS and the problem of finding minimum generalized eigenvalues of a parameterized matrix pencil. Finally, we present numerical results to illustrate this algorithm at the end of the paper.

On a constructive approach to optimality conditions for convex SIP problems with polyhedral index sets

In the paper, we consider a problem of convex Semi-Infinite Programming with an infinite index set in the form of a convex polyhedron. In study of this problem, we apply the approach suggested in our recent paper [Kostyukova OI, Tchemisova TV. Sufficient optimality conditions for convex Semi Infinite Programming. Optim. Methods Softw. 2010;25:279–297], and based on the notions of immobile indices and their immobility orders. The main result of the paper consists in explicit optimality conditions that do not use constraint qualifications and have the form of criterion. The comparison of the new optimality conditions with other known results is provided.

A new nonmonotone trust region method for unconstrained optimization equipped by an efficient adaptive radius

In this paper, we present a new nonmonotone trust region method with adaptive radius which is equipped by a nonmonotone line search technique for solving unconstrained optimization problems. The proposed method combines a modified version of the Li and Fukushima's nonmonotone technique in D.H. Li and M. Fukushima [A derivative-free line search and global convergence of Broyden like method for nonlinear equations, Optim. Methods Softw.13 (2000), pp. 181–201] for solving nonlinear systems with a new variant of Shi and Guo's adaptive strategy in Z.J. Shi and J.H. Guo [A new trust region methods for unconstrained optimization, J. Comput. Appl. Math. 213 (2008), pp. 509–520] for updating the trust region radius. The method performs a nonmonotone Armijo-type line search whenever the trial step is rejected. Under some standard assumptions, we provide the global convergence property as well as the superlinear and quadratic convergence rates for the new method. Numerical results show the efficiency and effectiveness of the new proposed method in practice.

An interesting characteristic of phase-1 of dual–primal algorithm for linear programming

Most algorithms for solving linear program require a phase-1 procedure to find a feasible solution. Recently, a dual–primal algorithm for linear optimization has been proposed by Li [Dual–primal algorithm for linear optimization, Optimiz. Methods Softw. 28 (2013), pp. 327–338]. In the process of implementing the dual–primal algorithms, we found an interesting phenomenon that the phase-1 algorithm developed in [Li (2013)] always terminates in one iteration. This fact does not come by chance. A rigorous proof is given in this paper.

A note about the complexity of minimizing Nesterov's smooth Chebyshev–Rosenbrock function

This short note considers and resolves the apparent contradiction between known worst-case complexity results for first- and second-order methods for solving unconstrained smooth nonconvex optimization problems and a recent note by Jarre [On Nesterov's smooth Chebyshev–Rosenbrock function, Optim. Methods Softw. (2011)] implying a very large lower bound on the number of iterations required to reach the solution's neighbourhood for a specific problem with variable dimension.

On an enumerative algorithm for solving eigenvalue complementarity problems

In this paper, we discuss the solution of linear and quadratic eigenvalue complementarity problems (EiCPs) using an enumerative algorithm of the type introduced by Judice et al. (Optim. Methods Softw. 24:549---586, 2009). Procedures for computing the interval that contains all the eigenvalues of the linear EiCP are first presented. A nonlinear programming (NLP) model for the quadratic EiCP is formulated next, and a necessary and sufficient condition for a stationary point of the NLP to be a solution of the quadratic EiCP is established. An extension of the enumerative algorithm for the quadratic EiCP is also developed, which solves this problem by computing a global minimum for the NLP formulation. Some computational experience is presented to highlight the efficiency and efficacy of the proposed enumerative algorithm for solving linear and quadratic EiCPs.

New complexity analysis of a Mehrotra-type predictor–corrector algorithm for semidefinite programming

In this paper, we propose a new Mehrotra-type predictor–corrector interior-point algorithm for semidefinite programming. This algorithm is an extension of the variant of Mehrotra-type algorithm that was proposed by Salahi et al. [On Mehrotra-type predictor–corrector algorithms, SIAM J. Optim. 18 (2007), pp. 1377–1397] for linear programming problems. We modify the step sizes lightly in the predictor step of Koulaei and Terlaky [On the complexity analysis of a Mehrotra-type primal–dual feasible algorithm for semidefinite optimization, Optim. Methods Softw. 25 (2010), pp. 467–485]. In such a way, we obtain O(nlog(Tr(X 0 S 0)/ϵ)) iteration complexity of the algorithm, where (X 0, y 0, S 0) is the initial feasible point and ϵ is the required precision.

A note on the global convergence theorem of the scaled conjugate gradient algorithms proposed by Andrei

In (Andrei, Comput. Optim. Appl. 38:402---416, 2007), the efficient scaled conjugate gradient algorithm SCALCG is proposed for solving unconstrained optimization problems. However, due to a wrong inequality used in (Andrei, Comput. Optim. Appl. 38:402---416, 2007) to show the sufficient descent property for the search directions of SCALCG, the proof of Theorem 2, the global convergence theorem of SCALCG, is incorrect. Here, in order to complete the proof of Theorem 2 in (Andrei, Comput. Optim. Appl. 38:402---416, 2007), we show that the search directions of SCALCG satisfy the sufficient descent condition. It is remarkable that the convergence analyses in (Andrei, Optim. Methods Softw. 22:561---571, 2007; Eur. J. Oper. Res. 204:410---420, 2010) should be revised similarly.

Convergence analysis of an extended auxiliary problem principle with various stopping criteria

We investigate the Extended Proximal Auxiliary Problem Principle (EPAPP) [A. Kaplan and R. Tichatschke, Extended Auxiliary Problem Principle using Bregman distances, Optimization 53 (2004), pp. 603–623, A. Kaplan and R. Tichatschke, Bregman functions and auxiliary problem principle, Optim. Methods Softw. 23, 1 (2008), pp. 95–107] for solving variational inequalities whose operator – here called the main operator – is the sum of a maximal monotone and a continuous operator. As usual for proximal methods using Bregman distances, Kalpan and Tichatschke required that the main operator is paramonotone. There are two main purposes of the present article. First, we establish the convergence of the EPAPP method without the use of paramonotonicity by replacing the latter assumption by other (partly rather natural) ones. An example showing that the main operator does not even have to be monotone is also given. Second, we discuss the EPAPP method with a stopping criterion of fixed-relative-error type by Solodov and Svaiter [An inexact hybrid generalized proximal point algorithm and some new results on the theory of Bregman functions, Math. Oper. Res. 25 (2000), pp. 214–230]. Kaplan and Tichatschke used a stopping criterion of summable-error type was used to admit an inexact solution of the corresponding auxiliary problems. To our knowledge, such a stopping criterion has not been applied to an extended APP yet; also in this situation, the paramonotonicity hypothesis is avoided. Independent of the stopping criterion under consideration, we admit an outer approximation of the set-valued component of the main operator. Owing to the use of Bregman-like functions to construct the symmetric components of the auxiliary operators, an interior-point effect is provided, meaning that – with a certain precaution – the auxiliary problems can be treated as unconstrained ones.

An efficient algorithm for solving rank one perturbed linear Diophantine systems using Rosser’s approach

Recently, we described a generalization of Rosser’s algorithm for a single linear Diophantine equation to an algorithm for solving systems of linear Diophantine equations. Here, we make use of the new formulation to present a new algorithm for solving rank one perturbed linear Diophantine systems, based on using Rosser’s approach. Finally, we compare the efficiency and effectiveness of our proposed algorithm with the algorithm proposed by Amini and Mahdavi-Amiri (Optim Methods Softw 21:819–831, 2006).

Properties of two DC algorithms in quadratic programming

Some new properties of the Projection DC decomposition algorithm (we call it Algorithm A) and the Proximal DC decomposition algorithm (we call it Algorithm B) Pham Dinh et al. in Optim Methods Softw, 23(4): 609---629 (2008) for solving the indefinite quadratic programming problem under linear constraints are proved in this paper. Among other things, we show that DCA sequences generated by Algorithm A converge to a locally unique solution if the initial points are taken from a neighborhood of it, and DCA sequences generated by either Algorithm A or Algorithm B are all bounded if a condition guaranteeing the solution existence of the given problem is satisfied.

Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity

An Adaptive Regularisation framework using Cubics (ARC) was proposed for unconstrained optimization and analysed in Cartis, Gould and Toint (Part I, Math Program, doi: 10.1007/s10107-009-0286-5, 2009), generalizing at the same time an unpublished method due to Griewank (Technical Report NA/12, 1981, DAMTP, University of Cambridge), an algorithm by Nesterov and Polyak (Math Program 108(1):177–205, 2006) and a proposal by Weiser, Deuflhard and Erdmann (Optim Methods Softw 22(3):413–431, 2007). In this companion paper, we further the analysis by providing worst-case global iteration complexity bounds for ARC and a second-order variant to achieve approximate first-order, and for the latter second-order, criticality of the iterates. In particular, the second-order ARC algorithm requires at most $${\mathcal{O}(\epsilon^{-3/2})}$$ iterations, or equivalently, function- and gradient-evaluations, to drive the norm of the gradient of the objective below the desired accuracy $${\epsilon}$$, and $${\mathcal{O}(\epsilon^{-3})}$$ iterations, to reach approximate nonnegative curvature in a subspace. The orders of these bounds match those proved for Algorithm 3.3 of Nesterov and Polyak which minimizes the cubic model globally on each iteration. Our approach is more general in that it allows the cubic model to be solved only approximately and may employ approximate Hessians.

Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results

An Adaptive Regularisation algorithm using Cubics (ARC) is proposed for unconstrained optimization, generalizing at the same time an unpublished method due to Griewank (Technical Report NA/12, 1981, DAMTP, University of Cambridge), an algorithm by Nesterov and Polyak (Math Program 108(1):177–205, 2006) and a proposal by Weiser et al. (Optim Methods Softw 22(3):413–431, 2007). At each iteration of our approach, an approximate global minimizer of a local cubic regularisation of the objective function is determined, and this ensures a significant improvement in the objective so long as the Hessian of the objective is locally Lipschitz continuous. The new method uses an adaptive estimation of the local Lipschitz constant and approximations to the global model-minimizer which remain computationally-viable even for large-scale problems. We show that the excellent global and local convergence properties obtained by Nesterov and Polyak are retained, and sometimes extended to a wider class of problems, by our ARC approach. Preliminary numerical experiments with small-scale test problems from the CUTEr set show encouraging performance of the ARC algorithm when compared to a basic trust-region implementation.

Accelerated conjugate gradient algorithm with finite difference Hessian/vector product approximation for unconstrained optimization

In this paper we propose a fundamentally different conjugate gradient method, in which the well-known parameter β k is computed by an approximation of the Hessian/vector product through finite differences. For search direction computation, the method uses a forward difference approximation to the Hessian/vector product in combination with a careful choice of the finite difference interval. For the step length computation we suggest an acceleration scheme able to improve the efficiency of the algorithm. Under common assumptions, the method is proved to be globally convergent. It is shown that for uniformly convex functions the convergence of the accelerated algorithm is still linear, but the reduction in function values is significantly improved. Numerical comparisons with conjugate gradient algorithms including CONMIN by Shanno and Phua [D.F. Shanno, K.H. Phua, Algorithm 500, minimization of unconstrained multivariate functions, ACM Trans. Math. Softw. 2 (1976) 87–94], SCALCG by Andrei [N. Andrei, Scaled conjugate gradient algorithms for unconstrained optimization, Comput. Optim. Appl. 38 (2007) 401–416; N. Andrei, Scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization, Optim. Methods Softw. 22 (2007) 561–571; N. Andrei, A scaled BFGS preconditioned conjugate gradient algorithm for unconstrained optimization, Appl. Math. Lett. 20 (2007) 645–650], and new conjugacy condition and related new conjugate gradient by Li, Tang and Wei [G. Li, C. Tang, Z. Wei, New conjugacy condition and related new conjugate gradient methods for unconstrained optimization, J. Comput. Appl. Math. 202 (2007) 523–539] or truncated Newton TN by Nash [S.G. Nash, Preconditioning of truncated-Newton methods, SIAM J. on Scientific and Statistical Computing 6 (1985) 599–616] using a set of 750 unconstrained optimization test problems show that the suggested algorithm outperforms these conjugate gradient algorithms as well as TN.

An improved contraction method for structured monotone variational inequalities

For solving monotone variational inequalities with separate structures, Ye and Yuan [A descent method for stuctured monotone variational inequalities, Optim. Methods Softw. 22 (2007), 329–338] used the iterates generated by the well-known alternating directions method to design a descent direction, and thus presented a contraction method. This article continues on this study. By observing an improved descent direction and, selecting the corresponding optimal step sizes, a new contraction method is presented. In addition to proving the algorithm's convergence under mild assumptions, we compare the improved contraction method to Ye and Yuan's method (which is generalized) and achieve the superiority of the new method in theoretical senses.

New approach for the nonlinear programming with transient stability constraints arising from power systems

This paper presents a new approach for solving a class of complicated nonlinear programming problems arises from optimal power flow with transient stability constraints (denoted by OTS) in power systems. By using a functional transformation technology proposed in Chen et al. (IEEE Trans. Circuits Syst. I Fundam. Theory Appl. 48:327---339, [2001]), the OTS problem is transformed to a semi-infinite programming (SIP). Then based on the KKT (Karush-Kuhn-Tucker) system of the reformulated SIP problem and the finite approximation technology, an iterative method is presented, which develops Wu-Li-Qi-Zhou' (Optim. Methods Softw. 20:629---643, [2005]) method. In order to save the computing cost, some typical computing technologies, such as active set strategy, quasi-Newton method for the subproblems coming from the finite approximation model, are addressed. The global convergence of the proposed algorithm is established. Numerical examples from power systems are tested. The computing results show the efficiency of the new approach.

An infeasible interior-point algorithm with full-Newton step for linear optimization

Recently, Roos (SIAM J Optim 16(4):1110–1136, 2006) presented a primal-dual infeasible interior-point algorithm that uses full-Newton steps and whose iteration bound coincides with the best known bound for infeasible interior-point algorithms. In the current paper we use a different feasibility step such that the definition of the feasibility step in Mansouri and Roos (Optim Methods Softw 22(3):519–530, 2007) is a special case of our definition, and show that the same result on the order of iteration complexity can be obtained.