Line Search Technique Research Articles

A New Machine Learning Algorithm Based on Optimization Method for Regression and Classification Problems

A convex minimization problem in the form of the sum of two proper lower-semicontinuous convex functions has received much attention from the community of optimization due to its broad applications to many disciplines, such as machine learning, regression and classification problems, image and signal processing, compressed sensing and optimal control. Many methods have been proposed to solve such problems but most of them take advantage of Lipschitz continuous assumption on the derivative of one function from the sum of them. In this work, we introduce a new accelerated algorithm for solving the mentioned convex minimization problem by using a linesearch technique together with a viscosity inertial forward–backward algorithm (VIFBA). A strong convergence result of the proposed method is obtained under some control conditions. As applications, we apply our proposed method to solve regression and classification problems by using an extreme learning machine model. Moreover, we show that our proposed algorithm has more efficiency and better convergence behavior than some algorithms mentioned in the literature.

A Two-Step Spectral Gradient Projection Method for System of Nonlinear Monotone Equations and Image Deblurring Problems

In this paper, we propose a two-step iterative algorithm based on projection technique for solving system of monotone nonlinear equations with convex constraints. The proposed two-step algorithm uses two search directions which are defined using the well-known Barzilai and Borwein (BB) spectral parameters.The BB spectral parameters can be viewed as the approximations of Jacobians with scalar multiple of identity matrices. If the Jacobians are close to symmetric matrices with clustered eigenvalues then the BB parameters are expected to behave nicely. We present a new line search technique for generating the separating hyperplane projection step of Solodov and Svaiter (1998) that generalizes the one used in most of the existing literature. We establish the convergence result of the algorithm under some suitable assumptions. Preliminary numerical experiments demonstrate the efficiency and computational advantage of the algorithm over some existing algorithms designed for solving similar problems. Finally, we apply the proposed algorithm to solve image deblurring problem.

A general alternative regularization method with line search technique for solving split equilibrium and fixed point problems in Hilbert spaces

In this paper, we introduce a new general alternative regularization algorithm for solving split equilibrium and fixed point problems in real Hilbert spaces. The proposed method does not require a prior estimate of the norm of the bounded linear operator nor a fixed stepsize for its convergence. Instead, we employ a line search technique and prove a strong convergence result for the sequence generated by the algorithm. A numerical experiment is given to show that the proposed method converges faster in terms of number of iteration and CPU time of computation than some existing methods in the literature.

The Least squares and line search in extracting eigenpairs in Jacobi–Davidson method

The methods used for extracting an approximate eigenpair are crucial in sparse iterative eigensolvers. Using least squares and line search techniques this paper devises a method for an approximate eigenpair extraction. Numerical comparison of the Jacobi–Davidson method using the suggested method of eigenpair extraction, Rayleigh–Ritz, and refined Ritz projections shows that the suggested method is a viable alternative.

A Filter and Nonmonotone Adaptive Trust Region Line Search Method for Unconstrained Optimization

In this paper, a new nonmonotone adaptive trust region algorithm is proposed for unconstrained optimization by combining a multidimensional filter and the Goldstein-type line search technique. A modified trust region ratio is presented which results in more reasonable consistency between the accurate model and the approximate model. When a trial step is rejected, we use a multidimensional filter to increase the likelihood that the trial step is accepted. If the trial step is still not successful with the filter, a nonmonotone Goldstein-type line search is used in the direction of the rejected trial step. The approximation of the Hessian matrix is updated by the modified Quasi-Newton formula (CBFGS). Under appropriate conditions, the proposed algorithm is globally convergent and superlinearly convergent. The new algorithm shows better performance in terms of the Dolan–Moré performance profile. Numerical results demonstrate the efficiency and robustness of the proposed algorithm for solving unconstrained optimization problems.

Unit stepsize for the Newton method close to critical solutions

As is well known, when initialized close to a nonsingular solution of a smooth nonlinear equation, the Newton method converges to this solution superlinearly. Moreover, the common Armijo linesearch procedure used to globalize the process for convergence from arbitrary starting points, accepts the unit stepsize asymptotically and ensures fast local convergence. In the case of a singular and possibly even nonisolated solution, the situation is much more complicated. Local linear convergence (with asymptotic ratio of 1/2) of the Newton method can still be guaranteed under reasonable assumptions, from a starlike, asymptotically dense set around the solution. Moreover, convergence can be accelerated by extrapolation and overrelaxation techniques. However, nothing was previously known on how the Newton method can be coupled in these circumstances with a linesearch technique for globalization that locally accepts unit stepsize and guarantees linear convergence. It turns out that this is a rather nontrivial issue, requiring a delicate combination of the analyses on acceptance of the unit stepsize and on the iterates staying within the relevant starlike domain of convergence. In addition to these analyses, numerical illustrations and comparisons are presented for the Newton method and the use of extrapolation to accelerate convergence speed.

A class of new derivative-free gradient type methods for large-scale nonlinear systems of monotone equations

In this paper, we present a class of new derivative-free gradient type methods for large-scale nonlinear systems of monotone equations. The methods combine the RMIL conjugate gradient method, the strategy of hyperplane projection and the derivative-free line search technique. Under some appropriate assumptions, the global convergence of the given methods is established. Numerical experiments indicate that the proposed methods are effective.

An improvement of adaptive cubic regularization method for unconstrained optimization problems

In this paper, we present two nonmonotone versions of adaptive cubic regularized (ARC) method for unconstrained optimization problems. The proposed methods are a combination of the ARC algorithm with the nonmonotone line search methods introduced by Zhang and Hager [A nonmonotone line search technique and its application to unconstrained optimization, SIAM J. Optim. 14 (2004), pp. 1043–1056] and Ahookhosh et al. [A nonmonotone trust-region line search method for large-scale unconstrained optimization, Appl. Math. Model. 36 (2012), pp. 478–487]. The global convergence analysis for these iterative algorithms is established under suitable conditions. Several numerical examples are given to illustrate the efficiency and robustness of the newly suggested methods. The obtained results show the satisfactory performance of the proposed algorithms when compared to the basic ARC algorithm.

Robust Resiliency-Oriented Operation of Active Distribution Networks Considering Windstorms

Recent climate changes have created intense natural disasters, such as windstorms, which can cause significant damages to power grids. System resilience is defined as the ability of the system to withstand such high-impact low-probability events. This paper proposes a robust resilient operational schedule for active distribution networks against windstorms. In order to capture dynamic behaviors of these disasters, zonal disaster-specific uncertainty sets associated with the windstorm are proposed. Additionally, the unavailability uncertainties of N-K contingencies as well as the forecast uncertainties of load demand, wind power, and solar power are taken into account. Instead of committing micro-turbines and energy storage systems in the first stage ( here-and-now ) of the decision-making process, the proposed model considers these commitment decisions in the second stage ( wait-and-see ) of the decision-making process, which is more consistent with the fast response time of these units. Since the second stage of the proposed model has binary decision variables, recent KKT-based and duality-based methods are not applicable. Therefore, a new solution method based on block coordinate descent (BCD) and line search (LS) techniques is proposed to solve the bi-level problem. Eventually, IEEE 33-bus distribution test system is used to illustrate the effectiveness of the proposed model and solution method.

A Riemannian Optimization Approach for Solving the Generalized Eigenvalue Problem for Nonsquare Matrix Pencils

In this paper, based on the Riemannian optimization approach we propose a Riemannian nonlinear conjugate gradient method with nonmonotone line search technique for solving the l parameterized original problem on generalized eigenvalue problems for nonsquare matrix pencils, which was first proposed by Chu and Golub (SIAM J Matrix Anal Appl 28:770–787, 2006). The new innovative approach is to reformulate the original optimization problem as a feasible optimization problem over a certain real product manifold. The global convergence of the proposed method is then established. Some numerical tests are given to demonstrate the efficiency of the proposed method. Comparisons with some latest methods are also given.

Convex relaxation for optimal rendezvous of unmanned aerial and ground vehicles

In this paper, we present an optimal rendezvous approach that facilitates the coordination of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). Of particular interest is the problem of UAV-UGV rendezvous for autonomous aerial refueling, which is an important capability for the future use of UAVs. The main contribution of this work is the development of a promising method for the generation of optimal rendezvous trajectories in real time using convex optimization. First, the UAV-UGV rendezvous problem is formulated as a nonconvex optimal control problem using an error dynamic model, where the state and control variables are highly coupled. Then, great effort is devoted to reducing the nonconvexity and coupling of the dynamics through change of variables. Based on a lossless convexification technique, a sequential convex programming algorithm is designed and the solution is obtained by solving a sequence of convex optimization problems. Theoretical proof is provided to demonstrate the exactness of convexification. Furthermore, a simple line search technique is introduced to handle the model error resulting from the linear approximations and to improve the convergence of the designed successive process. Simulation results of two refueling rendezvous situations validate that the proposed method is capable of converging to the solutions within a second, which shows great potential for real-time applications.

The PRP conjugate gradient algorithm with a modified WWP line search and its application in the image restoration problems

It is well known that the conjugate gradient algorithm is one of the most classic and useful methods for solving large-scale optimization problems, where the Polak-Ribière-Polyak(PRP) method is an important and effective conjugate gradient algorithm. However, the global convergence of the PRP conjugate gradient method can not be achieved when the WWP line search technique is used for a nonconvex function. In this paper, we propose a modified WWP line search technique for the PRP conjugate gradient algorithm. The new line search technique can guarantee the global convergence of the PRP method for general functions. The numerical results show that the PRP method with a new line search technique enables a practical computation and is effective in the image restoration problems.

Inertial extragradient method via viscosity approximation approach for solving equilibrium problem in Hilbert space

ABSTRACT In this paper, we propose a new viscosity type inertial extragradient method with Armijo line-search technique for approximating a common solution of equilibrium problem with pseudo-monotone bifunction and fixed points of relatively nonexpansive mapping in a real Hilbert space. Two advantages of our algorithm are that its convergence does not require the bifunction to satisfy any Lipschitz-type condition and only one strongly convex program and one projection onto the feasible set are perform at each iteration. Under some mild conditions on the control sequences, we state and prove a strong convergence theorem and also present two numerical examples to illustrate the performance of our algorithm. The results in this paper improve and generalize many recent results in this direction in the literature.

An alternating iterative algorithm for sound source identification based on equivalent source method

Near-field acoustical holography (NAH) based on equivalent source method (ESM) is an efficient technique for sound source identification. Conventional ESM with Tikhonov regularization (TRESM), ESM based on CVX MATLAB toolbox (CVX) and wideband acoustic holography (WBH) are commonly used methods for calculating equivalent source strengths. However, all of them have their respective limitations. To address some of these, an alternating iterative algorithm for sound source identification based on equivalent source method (AIESM) is proposed in this article, which is a combination of alternating direction method and a non-monotone line search technique. The method makes use of sparse regularization under the principle of compressive sensing (CS) to calculate equivalent source strengths. Moreover, inspired by the idea of functional beamforming (FB), AIESM with order n can yield an improved dynamic range when detecting the source location. Numerical simulations are carried out at different frequencies, and the results suggest that the computational efficiency of the proposed method is close to that of TRESM. In addition, AIESM has a better reconstruction accuracy than TRESM and WBH in a relatively wide frequency range. Compared with ESM based on CVX, AIESM is slightly better in reconstruction accuracy and has a higher computational efficiency. Meanwhile, AIESM with order n can provide more accurate source position and better resolution. The validity and practicality of the proposed method are further supported by experimental results.

A Robust Spectral Three-Term Conjugate Gradient Algorithm for Solving Unconstrained Minimization Problems

In this paper, we investigated a new improved Conjugate Gradient (CG) algorithm of a Three-Term type (TTCG) based on Dai and Liao procedure to improve the CG algorithm of (Hamoda, Rivaie, and Mamat / HRM). The new CG-algorithm satisfies both the conjugacy condition and the sufficient descent condition. The step-size of this TTCG-algorithm would be computed by accelerating the Wolfe-Powell line search technique. The proposed new TTCG algorithms have demonstrated their global affinity in certain specific circumstances given in this paper.

A robust algorithm for solving nonlinear system of equations using trust-region and line-search techniques

Newton's method is an attractive method for solving nonlinear system of equations because of its fast convergence property. However, Newton's method may fail if the Jacobian matrices are singular. Newton's method with trust-region can be used to avoid such problem. In this work, a new trust-region technique for Newton's method was formulated to solve the nonlinear system of equations. The search direction in this method is computed by a sequence of factorisations of the Jacobian matrix with modified structure using a Lagrange multiplier associated with trust-region constraint such that the final modified Jacobian turns out to be well-conditioned (regularised). An optimal Lagrange multiplier was deduced using the same idea of unconstrained optimisation to satisfy the trust-region constraint. Furthermore, Armijo line-search technique is integrated with the method in order to improve the step length. Numerical tests were conducted to investigate the performance of Newton's method integrated with trust-region and line-search techniques.

A robust algorithm for solving nonlinear system of equations using trust-region and line-search techniques

Newton's method is an attractive method for solving nonlinear system of equations because of its fast convergence property. However, Newton's method may fail if the Jacobian matrices are singular. Newton's method with trust-region can be used to avoid such problem. In this work, a new trust-region technique for Newton's method was formulated to solve the nonlinear system of equations. The search direction in this method is computed by a sequence of factorisations of the Jacobian matrix with modified structure using a Lagrange multiplier associated with trust-region constraint such that the final modified Jacobian turns out to be well-conditioned (regularised). An optimal Lagrange multiplier was deduced using the same idea of unconstrained optimisation to satisfy the trust-region constraint. Furthermore, Armijo line-search technique is integrated with the method in order to improve the step length. Numerical tests were conducted to investigate the performance of Newton's method integrated with trust-region and line-search techniques.

A Modified Spectral Gradient Projection Method for Solving Non-linear Monotone Equations with Convex Constraints and Its Application

In this paper, we propose a derivative free algorithm for solving non-linear monotone equations with convex constraints. The proposed algorithm combines the method of spectral gradient and the projection method. We also modify the backtracking line search technique. The global convergence of the proposed method is guaranteed, under the mild conditions. Further, the numerical experiments show that the large-scale non-linear equations with convex constraints can be effectively solved with our method. The $L_{1}$ -norm regularized problems in signal reconstruction are studied by using our method.

A new efficient hybrid conjugate gradient method based on LS-DY-HS conjugate gradient parameter

A nonlinear conjugate gradient method solves unconstrained optimisation problem based on an efficient line search technique and maintains a decent direction search (in case of a minimisation problem) with the help of conjugate gradient parameter. In this paper, a new hybrid conjugate gradient method based on a hybrid conjugate gradient parameter βk is proposed. The proposed βk combines linearly the conjugate gradient parameters of LS, DY and HS method. The present work also discusses the global convergence of the modified algorithm with inexact line search. Moreover, the proposed method is tested on the unconstrained problems from the library CUTEr (Gould et al., 2015) and the results have been compared with the other state of the art algorithms. The results in the numerical experiment show that the proposed hybrid algorithm is efficient.

Efficient matrix-free direction method with line search for solving large-scale system of nonlinear equations

We proposed a matrix-free direction with an inexact line search technique to solve system of nonlinear equations by using double direction approach. In this article, we approximated the Jacobian matrix by appropriately constructed matrix-free method via acceleration parameter. The global convergence of our method is established under mild conditions. Numerical comparisons reported in this paper are based on a set of large-scale test problems and show that the proposed method is efficient for large-scale problems.