Line Search Procedure Research Articles

A Novel Forward-Backward Algorithm for Solving Convex Minimization Problem in Hilbert Spaces

In this work, we aim to investigate the convex minimization problem of the sum of two objective functions. This optimization problem includes, in particular, image reconstruction and signal recovery. We then propose a new modified forward-backward splitting method without the assumption of the Lipschitz continuity of the gradient of functions by using the line search procedures. It is shown that the sequence generated by the proposed algorithm weakly converges to minimizers of the sum of two convex functions. We also provide some applications of the proposed method to compressed sensing in the frequency domain. The numerical reports show that our method has a better convergence behavior than other methods in terms of the number of iterations and CPU time. Moreover, the numerical results of the comparative analysis are also discussed to show the optimal choice of parameters in the line search.

Convergence of Inexact Forward--Backward Algorithms Using the Forward--Backward Envelope

This paper deals with a general framework for inexact forward--backward algorithms aimed at minimizing the sum of an analytic function and a lower semicontinuous, subanalytic, convex term. Such a framework relies on an implementable inexactness condition for the computation of the proximal operator and on a linesearch procedure, which is possibly performed whenever a variable metric is allowed into the forward--backward step. The main focus of this work is the convergence of the considered scheme without additional convexity assumptions on the objective function. Toward this aim, we employ the recent concept of forward--backward envelope to define a continuously differentiable surrogate function, which coincides with the objective at its stationary points and satisfies the so-called Kurdyka--Łojasiewicz (KL) property on its domain. We adapt the abstract convergence scheme usually exploited in the KL framework to our inexact forward--backward scheme, prove the convergence of the iterates to a stationary point of the problem, and prove the convergence rates for the function values. Finally, we show the effectiveness and the flexibility of the proposed framework on a large-scale image restoration test problem.

A Generalized Accelerated Composite Gradient Method: Uniting Nesterov's Fast Gradient Method and FISTA

The most popular first-order accelerated black-box methods for solving large-scale convex optimization problems are the Fast Gradient Method (FGM) and the Fast Iterative Shrinkage Thresholding Algorithm (FISTA). FGM requires that the objective be finite and differentiable with known gradient Lipschitz constant. FISTA is applicable to the more broad class of composite objectives and is equipped with a line-search procedure for estimating the Lipschitz constant. Nonetheless, FISTA cannot increase the step size and is unable to take advantage of strong convexity. FGM and FISTA are very similar in form. Despite this, they appear to have vastly differing convergence analyses. In this work we generalize the previously introduced augmented estimate sequence framework as well as the related notion of the gap sequence. We showcase the flexibility of our tools by constructing a Generalized Accelerated Composite Gradient Method, that unites FGM and FISTA, along with their most popular variants. The Lyapunov property of the generalized gap sequence used in deriving our method implies that both FGM and FISTA are amenable to a Lyapunov analysis, common among optimization algorithms. We further showcase the flexibility of our tools by endowing our method with monotonically decreasing objective function values alongside a versatile line-search procedure. By simultaneously incorporating the strengths of FGM and FISTA, our method is able to surpass both in terms of robustness and usability. We support our findings with simulation results on an extensive benchmark of composite problems. Our experiments show that monotonicity has a stabilizing effect on convergence and challenge the notion present in the literature that for strongly convex objectives, accelerated proximal schemes can be reduced to fixed momentum methods.

Subsampled Nonmonotone Spectral Gradient Methods

Abstract This paper deals with subsampled spectral gradient methods for minimizing finite sums. Subsample function and gradient approximations are employed in order to reduce the overall computational cost of the classical spectral gradient methods. The global convergence is enforced by a nonmonotone line search procedure. Global convergence is proved provided that functions and gradients are approximated with increasing accuracy. R-linear convergence and worst-case iteration complexity is investigated in case of strongly convex objective function. Numerical results on well known binary classification problems are given to show the effectiveness of this framework and analyze the effect of different spectral coefficient approximations arising from the variable sample nature of this procedure.

A Batch Variable Learning Rate Gradient Descent Algorithm With the SmoothingL1/2Regularization for Takagi-Sugeno Models

A batch variable learning rate gradient descent algorithm is proposed to efficiently train a neuro-fuzzy network of zero-order Takagi-Sugeno inference systems. By using the advantages of regularization, the smoothing L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sub> regularization is utilized to find more appropriate sparse network. Combining the second-order information of the smoothing error function, a variable learning rate is chosen along the steep descent direction, which avoids line search procedure and may reduce the cost of computation. In order to appropriately adjust the Lipschitz constant of the smoothing error function in the learning rate, a new scheme is proposed by introducing a hyper-parameter. Also the article applies the modified secant equation for estimating the Lipschitz constant, which makes the algorithm greatly reduce the oscillating phenomenon and improve the robustness. Under appropriate assumptions, a convergent result of the proposed algorithm is also given. Simulation results for two identification and classification problems show that the proposed algorithm has better numerical performance and promotes the sparsity capability of the network, compared with the common batch gradient descent algorithm and a variable learning rate gradient-based algorithm.

Descent Condition for a Scalar Parameter of Spectral HS (SpHS) Conjugate Gradient Method

Spectral conjugate gradient method has been used in most cases as an alternative to the conjugate gradient (CG) method in order to solve nonlinear unconstrained problems. In this paper, we introduced a spectral parameter of HS conjugate gradient method resultant from the classical CG search direction and used some of the standard test functions with numerous variables to prove its sufficient descent and global convergence properties, the numerical outcome is verified by exact line search procedures.

Weak convergence of explicit extragradient algorithms for solving equilibirum problems

This paper aims to propose two new algorithms that are developed by implementing inertial and subgradient techniques to solve the problem of pseudomonotone equilibrium problems. The weak convergence of these algorithms is well established based on standard assumptions of a cost bi-function. The advantage of these algorithms was that they did not need a line search procedure or any information on Lipschitz-type bifunction constants for step-size evaluation. A practical explanation for this is that they use a sequence of step-sizes that are updated at each iteration based on some previous iterations. For numerical examples, we discuss two well-known equilibrium models that assist our well-established convergence results, and we see that the suggested algorithm has a competitive advantage over time of execution and the number of iterations.

A Derivative-Free Decent Method Via Acceleration Parameter for Solving Systems of Nonlinear Equations

An algorithm for solving large-scale systems of nonlinear equations based on the transformation of the Newton method with the line search into a derivative-free descent method is introduced. Main idea used in the algorithm construction is to approximate the Jacobian by an appropriate diagonal matrix. Furthermore, the step length is calculated using inexact line search procedure. Under appropriate conditions, the proposed method is proved to be globally convergent under mild conditions. The numerical results presented show the efficiency of the proposed method.

Evaluation of anaerobic biodegradability potential and comparative kinetics of different agro-industrial substrates using a new hybrid computational coding scheme

Abstract The present analysis was conducted as the first study to investigate the biochemical methane potential of four different agro-industrial wastewaters originating from chocolate, slaughterhouse, gum, and beet sugar industries under the same anaerobic fermentation conditions. To the best of our knowledge, no previous study has specifically attempted to pinpoint a hybrid programming strategy for making a quantitative description of the anaerobic biodegradability of these waste streams. Thus, considering the scarcity of the literature in this field, a comprehensive study was conducted to evaluate the amount of bio-methane obtainable from the investigated organic wastes and to predict their kinetics using three different sigmoidal microbial growth curve models (modified Gompertz equation, transference function (reaction curve-type model), and logistic function) within the framework an original MATLAB®-based coding scheme. The results showed that methane productions started immediately after 4 h of incubation for all substrates and reached their maximum rates of 118, 116, 108, 34 mL CH4/g VS/day, respectively, for wastewaters from chocolate, slaughterhouse, gum, and beet sugar industries. The corrected mean steady state methane contents were 61.7%, 73.4%, 62.8%, and 62.1% in the respective order. The highest methane yield (943 mL CH4/g VS) was obtained from the slaughterhouse wastewater, and this value was 1.32, 1.58, and 4.56 times higher than those obtained in the anaerobic digestion of chocolate, gum, and beet sugar wastewaters, respectively. Among the three kinetic models tested, the logistic function best explained the behavior of the observed data of all substrates using a Quasi-Newton cubic line search procedure (R2 = 0.987–0.996) with minimum number of non-linear iterations and function counts. Deviations between the measured and the outputs of the best-fit kinetic model were less than 4.3% in prediction of methane production potentials, suggesting that the proposed computational methodology could be used as a well-suited and robust approach for modeling and optimization of a highly non-linear biosystem.

Subsampled inexact Newton methods for minimizing large sums of convex functions

Abstract This paper deals with the minimization of a large sum of convex functions by inexact Newton (IN) methods employing subsampled functions, gradients and Hessian approximations. The conjugate gradient method is used to compute the IN step and global convergence is enforced by a nonmonotone line-search procedure. The aim is to obtain methods with affordable costs and fast convergence. Assuming strongly convex functions, R-linear convergence and worst-case iteration complexity of the procedure are investigated when functions and gradients are approximated with increasing accuracy. A set of rules for the forcing parameters and subsample Hessian sizes are derived that ensure local q-linear/q-superlinear convergence of the proposed method. The random choice of the Hessian subsample is also considered and convergence in the mean square, both for finite and infinite sums of functions, is proved. Finally, the analysis of global convergence with asymptotic R-linear rate is extended to the case of the sum of convex functions and strongly convex objective function. Numerical results on well-known binary classification problems are also given. Adaptive strategies for selecting forcing terms and Hessian subsample size, streaming out of the theoretical analysis, are employed and the numerical results show that they yield effective IN methods.

Explicit Extragradient-Like Method with Regularization for Variational Inequalities

In this paper, we introduce and analyze the convergence of a new algorithm for solving a monotone and Lipschitz variational inequality problem in a Hilbert space. The algorithm uses variable stepsizes which are generated over each iteration, based on some previous iterates, and by some cheap computations. Contrary to many known algorithms, the resulting algorithm can be easily implemented without the prior knowledge of Lipschitz contant of operator, and also without any linesearch procedure. Besides, the regularization technique is suitably incorporated in the algorithm to get further convergence. Theorem of strong convergence is established under mild conditions imposed on control parameters. Some experiments are provided to illustrate the numerical behavior of the algorithm in comparison with others.

Modified extragradient-like algorithms with new stepsizes for variational inequalities

The paper concerns with an algorithm for approximating solutions of a variational inequality problem involving a Lipschitz continuous and monotone operator in a Hilbert space. The algorithm uses a new stepsize rule which does not depend on the Lipschitz constant and without any linesearch procedure. The resulting algorithm only requires to compute a projection on feasible set and a value of operator over each iteration. The convergence and the convergence rate of the algorithm are established. Some experiments are performed to show the numerical behavior of the proposed algorithm and also to compare its performance with those of others.

Volume-based phase stability analysis including capillary pressure

The effect of capillary pressure on phase equilibrium is very important in tight formations, such as shale oil and gas reservoirs. In confined fluids, the capillary pressure induced by highly curved interfaces causes bubble points suppression and an inflated dew point locus, with a shift of cricondentherm points towards higher temperatures, as compared to the bulk fluid. In the conventional phase stability testing including capillary pressure, the problem is solved using the classical tangent plane distance (TPD) function (related to the Gibbs free energy surface) and mole numbers as primary variables. In this work, a volume-based approach (in which the equation of state needs not to be solved for volume) is used to solve this problem, as a bound and linear inequality constrained minimization of a TPD function with respect to the component molar densities. A modified Cholesky factorization (to ensure a descent direction) and a two-stage line search procedure (to ensure that iterates remain in the feasible domain and the objective function is decreased) are used in Newton iterations; a proper change of variables strengthens the robustness. The Weinaug-Katz equation (widely used in chemical and petroleum industry), which is a function of molar densities only, is used for interfacial tensions; the additional partial derivatives in the gradient vector and Hessian matrix have very simple expressions, unlike in conventional formulations. The proposed method is tested for two hydrocarbon mixtures (an oil and a gas condensate) in a wide range of pressures, temperatures and curvature radii, with a special attention paid to the convergence behavior near the singularities, that is, the stability test limit locus (STLL) and the spinodal. The method proves to be highly robust and exhibits fast convergence, showing a similar convergence behavior and almost the same computational effort as in the case of a bulk fluid. The results are only slightly different from conventional methods (except at high curvatures near the cricondentherm and at low pressures); differences are due to the fact that the dependence of capillary pressure on composition is taken into account in deriving the stationarity conditions, unlike in the conventional approach. The proposed method is not dependent on the thermodynamic model and can be used with any capillary pressure representation in which the interfacial tension model is explicit in molar densities.

Explicit iterative algorithms for solving equilibrium problems

The paper introduces two new extragradient algorithms for solving a pseudomonotone equilibrium problem with a Lipschitz-type condition recently presented by Mastroeni in auxiliary problem principle. The algorithm uses variable stepsizes which are updated at each iteration and based on some previous iterates. The advantage of the algorithms is that they are done without the prior knowledge of Lipschitz-type constants and also without any linesearch procedure. The convergence of the algorithms is established under mild assumptions. In the case where the equilibrium bifunction is strongly pseudomonotone, the R-linear rate of convergence of the new algorithms is formulated. Several of fundamental experiments are provided to illustrate the numerical behavior of the algorithms and also to compare with others.

An adaptive nonmonotone truncated Newton method for optimal control of a class of parabolic distributed parameter systems

A black-box method using the finite elements, the Crank–Nicolson and a nonmonotone truncated Newton (TN) method is presented for solving optimal control problems (OCPs) governed by partial differential equations (PDEs). The proposed method finds the optimal control of a class of linear and nonlinear parabolic distributed parameter systems with a quadratic cost functional. To this end, the piecewise linear finite elements method and the well-known Crank–Nicolson method are used for discretizing in space and in time, respectively. Afterwards, regarding the implicit function theorem (IFT), the optimal control problem is transformed into an unconstrained nonlinear optimization problem. Considering that in a gradient-based method for solving optimal control problems, the evaluations of gradients and Hessians of the cost functional is important, hence, an adjoint technique is used to evaluate them effectively. In addition, to make a globalization strategy, we first introduce an adaptive nonmonotone strategy which properly controls the degree of nonmonotonicity and then incorporate it into an inexact Armijo-type line search approach to construct a more relaxed line search procedure. Finally, the obtained unconstrained nonlinear optimization problem is solved by utilizing the proposed nonmonotone truncated Newton method. Results gained from the new offered method compared with existing methods show that the new method is promising.

Accelerated Primal-Dual Gradient Descent with Linesearch for Convex, Nonconvex, and Nonsmooth Optimization Problems

A new version of accelerated gradient descent is proposed. The method does not require any a priori information on the objective function, uses a linesearch procedure for convergence acceleration in practice, converge according to well-known lower bounds for both convex and nonconvex objective functions, and has primal-dual properties. A universal version of this method is also described.

Improved quasi-newton method via SR1 update for solving symmetric systems of nonlinear equations

The systems of nonlinear equations emerges from many areas of computing, scientific and engineering research applications. A variety of an iterative methods for solving such systems have been developed, this include the famous Newton method. Unfortunately, the Newton method suffers setback, which includes storing matrix at each iteration and computing Jacobian matrix, which may be difficult or even impossible to compute. To overcome the drawbacks that bedeviling Newton method, a modification to SR1 update was proposed in this study. With the aid of inexact line search procedure by Li and Fukushima, the modification was achieved by simply approximating the inverse Hessian matrix with an identity matrix without computing the Jacobian. Unlike the classical SR1 method, the modification neither require storing matrix at each iteration nor needed to compute the Jacobian matrix. In finding the solution to non-linear problems of the form 40 benchmark test problems were solved. A comparison was made with other two methods based on CPU time and number of iterations. In this study, the proposed method solved 37 problems effectively in terms of number of iterations. In terms of CPU time, the proposed method also outperformed the existing methods. The contribution from the methodology yielded a method that is suitable for solving symmetric systems of nonlinear equations. The derivative-free feature of the proposed method gave its advantage to solve relatively large-scale problems (10,000 variables) compared to the existing methods. From the preliminary numerical results, the proposed method turned out to be significantly faster, effective and suitable for solving large scale symmetric nonlinear equations.

Two–parameter scaled memoryless BFGS methods with a nonmonotone choice for the initial step length

A class of two–parameter scaled memoryless BFGS methods is developed for solving unconstrained optimization problems. Then, the scaling parameters are determined in a way to improve the condition number of the corresponding memoryless BFGS update. It is shown that for uniformly convex objective functions, search directions of the method satisfy the sufficient descent condition which leads to the global convergence. To achieve convergence for general functions, a revised version of the method is developed based on the Li–Fukushima modified secant equation. To enhance performance of the methods, a nonmonotone scheme for computing the initial value of the step length is suggested to be used in the line search procedure. Numerical experiments are done on a set of unconstrained optimization test problems of the CUTEr collection. They show efficiency of the proposed algorithms in the sense of the Dolan–More performance profile.

A universal modification of the linear coupling method

ABSTRACTIn the late sixties, N. Shor and B. Polyak independently proposed optimal first-order methods for solving non-smooth convex optimization problems. In 1982 A. Nemirovski proposed optimal first-order methods for solving smooth convex optimization problems, which utilized auxiliary line search. In 1985 A. Nemirovski and Yu. Nesterov proposed a parametric family of optimal first-order methods for solving convex optimization problems with intermediate smoothness. In 2013 Yu. Nesterov proposed a universal gradient method which combined all good properties of the previous methods, except the possibility of using auxiliary line search. One can typically observe that in practice auxiliary line search improves performance for many tasks. In this paper, we propose the apparently first such method of non-smooth convex optimization allowing the use of the line search procedure. Moreover, it is based on the universal gradient method, which does not require any a priori information about the actual degree of smoothness of the problem. Numerical experiments demonstrate that the proposed method is, in some cases, considerably faster than Nesterov's universal gradient method.

INITIAL IMPROVEMENT OF THE HYBRID ACCELERATED GRADIENT DESCENT PROCESS

We improve the convergence properties of the iterative scheme for solving unconstrained optimisation problems introduced in Petrovic et al. [‘Hybridization of accelerated gradient descent method’, Numer. Algorithms (2017), doi:10.1007/s11075-017-0460-4] by optimising the value of the initial step length parameter in the backtracking line search procedure. We prove the validity of the algorithm and illustrate its advantages by numerical experiments and comparisons.