Step-size Rule Research Articles

New iterative regularization methods for solving split variational inclusion problems

<p style='text-indent:20px;'>The paper proposes some new iterative algorithms for solving a split variational inclusion problem involving maximally monotone multi-valued operators in a Hilbert space. The algorithms are constructed around the resolvent of operator and the regularization technique to get the strong convergence. Some stepsize rules are incorporated to allow the algorithms to work easily. An application of the proposed algorithms to split feasibility problems is also studied. The computational performance of the new algorithms in comparison with others is shown by some numerical experiments.</p>

A modified self-adaptive dual ascent method with relaxed stepsize condition for linearly constrained quadratic convex optimization

<p style='text-indent:20px;'>Dual ascent method (DAM) is an effective method for solving linearly constrained convex optimization problems. Classical DAM converges extremely slowly due to its small stepsize, and it has been improved through relaxing the stepsize condition and introducing the self-adaptive stepsize rule by He et al, which increases its convergence speed. In this paper, we further relax its stepsize condition whereas the convergence result can still be guaranteed, providing the objective function is quadratic. We show the encouraging performance of the new DAM with new stepsize condition via the experiments on both synthetic and real problems.</p>

Revisiting the Curvature-aided IAG: Improved Theory and Reduced Complexity

The curvature-aided IAG (CIAG) algorithm is an efficient asynchronous optimization method that accelerates IAG using a delay compensation technique. However, existing step-size rules for CIAG are conservative and hard to implement, and the Hessian computation in CIAG is often computationally expensive. To alleviate these issues, we first provide an easy-to-implement and less conservative step-size rule for CIAG. Next, we propose a modified CIAG algorithm that reduces the computational complexity by approximating the Hessian with a constant matrix. Convergence results are derived for each algorithm on both convex and strongly convex problems, and numerical experiments on logistic regression demonstrate their effectiveness in practice.

A novel class of forward-backward explicit iterative algorithms using inertial techniques to solve variational inequality problems with quasi-monotone operators

<abstract><p>The theory of variational inequalities is an important tool in physics, engineering, finance, and optimization theory. The projection algorithm and its variants are useful tools for determining the approximate solution to the variational inequality problem. This paper introduces three distinct extragradient algorithms for dealing with variational inequality problems involving quasi-monotone and semistrictly quasi-monotone operators in infinite-dimensional real Hilbert spaces. This problem is a general mathematical model that incorporates a set of applied mathematical models as an example, such as equilibrium models, optimization problems, fixed point problems, saddle point problems, and Nash equilibrium point problems. The proposed algorithms employ both fixed and variable stepsize rules that are iteratively transformed based on previous iterations. These algorithms are based on the fact that no prior knowledge of the Lipschitz constant or any line-search framework is required. To demonstrate the convergence of the proposed algorithms, some simple conditions are used. Numerous experiments have been conducted to highlight the numerical capabilities of algorithms.</p></abstract>

Incremental subgradient algorithms with dynamic step sizes for separable convex optimizations

We consider the incremental subgradient algorithm employing dynamic step sizes for minimizing the sum of a large number of component convex functions. The dynamic step size rule was firstly introduced by Goffin and Kiwiel [Math. Program., 1999, 85(1): 207‐211] for the subgradient algorithm, soon later, for the incremental subgradient algorithm by Nedic and Bertsekas in [SIAM J. Optim., 2001, 12(1): 109‐138]. It was observed experimentally that the incremental approach has been very successful in solving large separable optimizations and that the dynamic step sizes generally have better computational performance than others in the literature. In the present paper, we propose two modified dynamic step size rules for the incremental subgradient algorithm and analyse the convergence and complexity properties of them. At last, the assignment problem is considered and the incremental subgradient algorithms employing different kinds of dynamic step sizes are applied to solve the problem. The computational experiments show that the two modified ones converges dramatically faster and more stable than the corresponding one in [SIAM J. Optim., 2001, 12(1): 109‐138]. Particularly, for solving large separable convex optimizations, we strongly recommend the second one (see Algorithm 3.3 in the paper) since it has interesting computational performance and is the simplest one.

Inertial subgradient extragradient method for solving pseudomonotone equilibrium problems and fixed point problems in Hilbert spaces

This paper proposes a new inertial subgradient extragradient method for solving equilibrium problems with pseudomonotone and Lipschitz-type bifunctions and fixed point problems for nonexpansive mappings in real Hilbert spaces. Precisely, we prove that the sequence generated by proposed algorithm converges strongly to a common solution of equilibrium problems and fixed point problems. We use an effective self-adaptive step size rule to accelerate the convergence process of our proposed iterative algorithm. Moreover, some numerical results are given to show the effectiveness of the proposed algorithm. The results obtained in this paper extend and improve many recent ones in the literature.

Strong convergence analysis of modified Mann-type forward–backward scheme for solving quasimonotone variational inequalities

The paper proposes multiple new extragradient methods for solving a variational inequality problem involving quasimonotone operators in infinite-dimensional real Hilbert spaces. These methods contain variable stepsize rules that are revised at each iteration and are dependent on prior iterations. These algorithms have the benefit of not requiring prior knowledge of the Lipschitz constant or any line-search approach. Simple conditions are used to demonstrate the algorithm’s convergence. A collection of simple experiments is presented to show the numerical behavior of the algorithms.

On solving pseudomonotone equilibrium problems via two new extragradient-type methods under convex constraints

Abstract The primary objective of this study is to develop two new proximal-type algorithms for solving equilibrium problems in real Hilbert space. Both new algorithms are analogous to the well-known two-step extragradient algorithm for solving the variational inequality problem in Hilbert spaces. The proposed iterative algorithms use a new step size rule based on local bifunction information instead of the line search technique. Two weak convergence theorems for both algorithms are well-established by letting mild conditions. The main results are used to solve the fixed point and variational inequality problems. Finally, we present several computational experiments to demonstrate the efficiency and effectiveness of the proposed algorithms.

Regularization Method for the Variational Inequality Problem over the Set of Solutions to the Generalized Equilibrium Problem

The paper is devoted to bilevel problems: variational inequality problems over the set of solutions to the generalized equilibrium problems in a Hilbert space. To solve these problems, an iterative algorithm is proposed that combines the ideas of the Tseng’s extragradient method, the inertial idea and iterative regularization. The proposed method adopts a non-monotonic stepsize rule without any line search procedure. Under suitable conditions, the strong convergence of the resulting method is obtained. Several numerical experiments are also provided to illustrate the efficiency of the proposed method with respect to certain existing ones.

Modified Inertial Subgradient Extragradient Method with Regularization for Variational Inequality and Null Point Problems

The paper develops a modified inertial subgradient extragradient method to find a solution to the variational inequality problem over the set of common solutions to the variational inequality and null point problems. The proposed method adopts a nonmonotonic stepsize rule without any linesearch procedure. We describe how to incorporate the regularization technique and the subgradient extragradient method; then, we establish the strong convergence of the proposed method under some appropriate conditions. Several numerical experiments are also provided to verify the efficiency of the introduced method with respect to previous methods.

Convergence Analysis of New Construction Explicit Methods for Solving Equilibrium Programming and Fixed Point Problems

In this paper, we present improved iterative methods for evaluating the numerical solution of an equilibrium problem in a Hilbert space with a pseudomonotone and a Lipschitz-type bifunction. The method is built around two computing phases of a proximal-like mapping with inertial terms. Many such simpler step size rules that do not involve line search are examined, allowing the technique to be enforced more effectively without knowledge of the Lipschitz-type constant of the cost bifunction. When the control parameter conditions are properly defined, the iterative sequences converge weakly on a particular solution to the problem. We provide weak convergence theorems without knowing the Lipschitz-type bifunction constants. A few numerical tests were performed, and the results demonstrated the appropriateness and rapid convergence of the new methods over traditional ones.

Inertial Modification Using Self-Adaptive Subgradient Extragradient Techniques for Equilibrium Programming Applied to Variational Inequalities and Fixed-Point Problems

Equilibrium problems are articulated in a variety of mathematical computing applications, including minimax and numerical programming, saddle-point problems, fixed-point problems, and variational inequalities. In this paper, we introduce improved iterative techniques for evaluating the numerical solution of an equilibrium problem in a Hilbert space with a pseudomonotone and a Lipschitz-type bifunction. These techniques are based on two computing steps of a proximal-like mapping with inertial terms. We investigated two simplified stepsize rules that do not require a line search, allowing the technique to be carried out more successfully without knowledge of the Lipschitz-type constant of the cost bifunction. Once control parameter constraints are put in place, the iterative sequences converge on a particular solution to the problem. We prove strong convergence theorems without knowing the Lipschitz-type bifunction constants. A sequence of numerical tests was performed, and the results confirmed the correctness and speedy convergence of the new techniques over the traditional ones.

A stochastic gradient descent approach with partitioned-truncated singular value decomposition for large-scale inverse problems of magnetic modulus data

We propose a stochastic gradient descent approach with partitioned-truncated singular value decomposition (SVD) for large-scale inverse problems of magnetic modulus data. Motivated by a uniqueness theorem in gravity inverse problem and realizing the similarity between gravity and magnetic inverse problems, we propose to solve the level-set function modeling the volume susceptibility distribution from the nonlinear magnetic modulus data. To deal with large-scale data, we employ a mini-batch stochastic gradient descent approach with random reshuffling when solving the optimization problem of the inverse problem. We propose a stepsize rule for the stochastic gradient descent according to the Courant–Friedrichs–Lewy condition of the evolution equation. In addition, we develop a partitioned-truncated SVD algorithm for the linear part of the inverse problem in the context of stochastic gradient descent. Numerical examples illustrate the efficacy of the proposed method, which turns out to have the capability of efficiently processing large-scale measurement data for the magnetic inverse problem. A possible generalization to the inverse problem of deep neural network is discussed at the end.

The inertial iterative extragradient methods for solving pseudomonotone equilibrium programming in Hilbert spaces

In this paper, we present new iterative techniques for approximating the solution of an equilibrium problem involving a pseudomonotone and a Lipschitz-type bifunction in Hilbert spaces. These techniques consist of two computing steps of a proximal-type mapping with an inertial term. Improved simplified stepsize rules that do not involve line search are investigated, allowing the method to be implemented more quickly without knowing the Lipschitz-type constants of a bifunction. The iterative sequences converge weakly on a specific solution to the problem when the control parameter conditions are properly specified. The numerical tests were carried out, and the results demonstrated the applicability and quick convergence of innovative approaches over earlier ones.

F$$_3$$ORNITS : a flexible variable step size non-iterative co-simulation method handling subsystems with hybrid advanced capabilities

This paper introduces the F $$_3$$ ORNITS non-iterative co-simulation algorithm in which F $$_3$$ stands for the 3 flexible aspects of the method: flexible polynomial order representation of coupling variables, flexible time-stepper applying variable co-simulation step size rules on subsystems allowing it, and flexible scheduler orchestrating the meeting times among the subsystems and capable of asynchronousness when subsystems’ constraints require it. The motivation of the F $$_3$$ ORNITS method is to accept any kind of co-simulation model as far as they represent circuits (0D models, such as ODE or DAE), including any kind of subsystem (open circuits), regardless on their available capabilities. Indeed, one of the major problems in industry is that the subsystems usually have constraints or lack of advanced capabilities making it impossible to implement most of the advanced co-simulation algorithms on them. The method makes it possible to preserve the dynamics of the coupling constraints when necessary as well as to avoid breaking $$C^1$$ smoothness at communication times, and also to adapt the co-simulation step size in a way that is robust both to zero-crossing variables (contrary to classical relative error-based criteria) and to jumps. Five test cases are presented to illustrate the robustness of the F $$_3$$ ORNITS method as well as its higher accuracy than the non-iterative Jacobi coupling algorithm (the most commonly used method in industry) for a smaller number of co-simulation steps.

A new class of inertial algorithms with monotonic step sizes for solving fixed point and variational inequalities

This study presents two inertial type extragradient algorithms for finding a common solution to the monotone variational inequalities and fixed point problems for ‐demicontractive mapping in real Hilbert spaces. We provide inertial type iterative algorithms with self‐adaptive variable step size rules that do not require prior knowledge of the operator value. Our algorithms employ a basic step size rule, which is derived by certain computations at each iteration. Without previous knowledge of the operators Lipschitz constant, two strong convergence theorems were obtained. Finally, we present a number of numerical experiments to evaluate the efficacy and applicability of the proposed algorithms. The conclusions of this study on variational inequality and fixed point problems support and extend previous findings.

A New Construction and Convergence Analysis of Non-Monotonic Iterative Methods for Solving ρ-Demicontractive Fixed Point Problems and Variational Inequalities Involving Pseudomonotone Mapping

Two new inertial-type extragradient methods are proposed to find a numerical common solution to the variational inequality problem involving a pseudomonotone and Lipschitz continuous operator, as well as the fixed point problem in real Hilbert spaces with a ρ-demicontractive mapping. These inertial-type iterative methods use self-adaptive step size rules that do not require previous knowledge of the Lipschitz constant. We also show that the proposed methods strongly converge to a solution of the variational inequality and fixed point problems under appropriate standard test conditions. Finally, we present several numerical examples to show the effectiveness and validation of the proposed methods.

On solving variational inequality problems involving quasimonotone operators via modified Tseng's extragradient methods with convergence analysis

The main objective of this research is to find the numerical solution of variational inequalities involving quasimonotone operators in infinite-dimensional real Hilbert spaces. The main advantage of these iterative schemes is that they allow the uncomplicated calculation of step size rules that depend on the knowledge of an operator explanation instead of the Lipschitz constant or some other line search method. The proposed iterative schemes follow a monotone and non-monotone step size procedure based on mapping (operator) information as a replacement for its Lipschitz constant or some other line search method. The strong convergences are well proven, analogous to the proposed methods, and impose certain control specification conditions. Finally, to verify the effectiveness of the iterative methods, we present some numerical experiments.

Improved variable step-size least mean square algorithm based on Sigmoid function

One of the research focuses on the least mean square (LMS) algorithm is how to design the variable step-size rule to make the LMS algorithm converge ratio and steady-state error. The step-size is relatively large in its early stages; that is, the algorithm at this stage converges quickly; when the algorithm tends to a steady-state, the step-size is relatively small, that is. It is at this stage that the steady-state estimation error is relatively tiny. Therefore, looking at it as a whole, the algorithm can converge quickly with lower steady-state errors. Therefore, this paper designs a new step-size rule based on the Sigmoid function and theoretically analyzes the convergence characteristics and steady-state performance. Experiments and simulation comparisons are carried out under the conditions of comparison. The theoretical analysis combined with experimental simulation verification: even if the linear system has a sudden change, this algorithm still has a faster convergence ratio and tracking speed and can obtain minor steady-state errors and steady-state offsets.

Two Nonmonotonic Self-Adaptive Strongly Convergent Projection-Type Methods for Solving Pseudomonotone Variational Inequalities

The primary objective of this study is to introduce two novel extragradient-type iterative schemes for solving variational inequality problems in a real Hilbert space. The proposed iterative schemes extend the well-known subgradient extragradient method and are used to solve variational inequalities involving the pseudomonotone operator in real Hilbert spaces. The proposed iterative methods have the primary advantage of using a simple mathematical formula for step size rule based on operator information rather than the Lipschitz constant or another line search method. Strong convergence results for the suggested iterative algorithms are well-established for mild conditions, such as Lipschitz continuity and mapping monotonicity. Finally, we present many numerical experiments that show the effectiveness and superiority of iterative methods.