Conditions For Global Convergence Research Articles

Convergence and complexity guarantees for a wide class of descent algorithms in nonconvex multi-objective optimization

We address conditions for global convergence and worst-case complexity bounds of descent algorithms in nonconvex multi-objective optimization. Specifically, we define the concept of steepest-descent-related directions. We consider iterative algorithms taking steps along such directions, selecting the stepsize according to a standard Armijo-type rule. We prove that methods fitting this framework automatically enjoy global convergence properties. Moreover, we show that a slightly stricter property, satisfied by most known algorithms, guarantees the same complexity bound of O(ϵ−2) as the steepest descent method.

Local conditions for global convergence of gradient flows and proximal point sequences in metric spaces

This paper deals with local criteria for the convergence to a global minimiser for gradient flow trajectories and their discretisations. To obtain quantitative estimates on the speed of convergence, we consider variations on the classical Kurdyka–Łojasiewicz inequality for a large class of parameter functions. Our assumptions are given in terms of the initial data, without any reference to an equilibrium point. The main results are convergence statements for gradient flow curves and proximal point sequences to a global minimiser, together with sharp quantitative estimates on the speed of convergence. These convergence results apply in the general setting of lower semicontinuous functionals on complete metric spaces, generalising recent results for smooth functionals on R n \mathbb {R}^n . While the non-smooth setting covers very general spaces, it is also useful for (non)-smooth functionals on R n \mathbb {R}^n .

Global stability of delayed genetic regulatory networks with wider hill functions: A mixing monotone semiflows approach

In this paper, we study the global stability of delayed genetic regulatory networks (DGRNs) with Hill-type activation (or inhibition) functions based on the mixing monotone semiflows approach, where Hill coefficients can be arbitrary positive real number. A new result on the global stability of DGRN is given in the case that all Hill coefficients are less than or equal to 1. In addition, for all Hill coefficients greater than 1, a new sufficient condition for global convergence of DGRN is given, which is less conservative than the condition of the existing correlation results. Finally, two numerical examples and simulations are given to explain the effect of obtained results.

A framework using nested partitions algorithm for convergence analysis of population distribution-based methods

Stochastic optimization algorithms such as genetic algorithm (GA), particle swarm optimization (PSO), estimation of distribution algorithms (EDAs), and nested partitions algorithm (NPA) are used in many problems including nonlinear model predictive control and task assignment. Some of these algorithms, however, lack global convergence guarantee such as PSO, or require strict convergence assumptions such as NPA. To enhance these methods in terms of convergence, a common underlying framework towards representing the seemingly unrelated methods is established as the updating of the distribution of the population through iterative sampling, and the methods that fit into this framework are called population distribution-based methods. Global convergence conditions for this framework are innovatively developed by building a shadow NPA structure for the population evolution process. The result is generic and is capable of analyzing convergence of many methods including GA, PSO, EDA, and NPA. It can be further exploited to improve convergence by modifying these methods. The existing and modified variants of these methods are then applied to case studies to show the improvement.

Modified three-term conjugate gradient algorithm and its applications in image restoration

In image restoration, the goal is often to bring back a high-quality version of an image from a lower-quality copy of it. In this article, we will investigate one kind of recovery issue, namely recovering photos that have been blurred by noise in digital photographs (sometimes known as "salt and pepper" noise). When subjected to noise at varying frequencies and intensities (30,50,70,90). In this paper, we used the conjugate gradient algorithm to Restorative images and remove noise from them, we developed the conjugate gradient algorithm with three limits using the conjugate condition of Dai and Liao, where the new algorithm achieved the conditions for descent and global convergence under some assumptions. According to the results of the numerical analysis, the recently created approach is unequivocally superior to both the fletcher and reeves (FR) method and the fletcher and reeves three-term (TTFR) metod. Use the structural similarity index measure (SSIM), which is used to measure image quality and the higher its value, the better the result. The original image was compared with all the noisy images and each according to the percentage of noise as well as the images processed with the four methods.

Global stabilization of uncertain nonlinear systems via fractional-order PID

This work presents a method that analyzes the global stabilization of fractional-order uncertain nonlinear feedback systems classes with fractional-order proportional–integral–derivative (PID) controllers. Two theorems are provided to necessary conditions for global convergence to any desired setpoints by designing controllers. The first theorem addresses a class of second-order time-varying systems controlled by fractional-order PID controllers, which extends the main result about PID (Zhao and Guo, 2017) into fractional-order systems via different analysis methods. The second theorem investigates another class of first-order time-invariant systems regulated by fractional-order proportional–integral (PI) controllers. The method is illustrated on two feedback systems with controllers to ensure the global convergence of the feedback system to desired setpoints.

Diffusion-Based Distributed Parameter Estimation Through Directed Graphs With Switching Topology: Application of Dynamic Regressor Extension and Mixing

In this article, we consider the problem of discrete-time, diffusion-based distributed parameter estimation with the agents connected via <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">directed</i> graphs with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">switching</i> topologies and a self loop at each node. We show that, by incorporating the recently introduced dynamic regressor extension and mixing procedure to a classical gradient-descent algorithm, improved convergence properties can be achieved. In particular, it is shown that with this modification sufficient conditions for global convergence of all the estimators is that one of the sensors receives enough information to generate a consistent estimate and that this sensor is “well-connected.” The main feature of this result is that the excitation condition imposed on this distinguished sensor is strictly weaker than the classical persistent excitation requirement. The connectivity assumption is also very mild, requiring only that the union of the edges of all connectivity graphs over any time interval with an arbitrary but fixed length contains a spanning tree rooted at the information-rich node. In the case of nonswitching topologies, this assumption is satisfied by strongly connected graphs, and not only by them.

Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach

AbstractNeural networks, a central tool in machine learning, have demonstrated remarkable, high fidelity performance on image recognition and classification tasks. These successes evince an ability to accurately represent high‐dimensional functions, but rigorous results about the approximation error of neural networks after training are few. Here we establish conditions for global convergence of the standard optimization algorithm used in machine learning applications, stochastic gradient descent (SGD), and quantifying the scaling of its error with the size of the network. This is done by reinterpreting SGD as the evolution of a particle system with interactions governed by a potential related to the objective or “loss” function used to train the network. We show that, when the number n of units is large, the empirical distribution of the particles descends on a convex landscape towards the global minimum at a rate independent of n, with a resulting approximation error that universally scales as O(n−1). These properties are established in the form of a law of large numbers and a central limit theorem for the empirical distribution. Our analysis also quantifies the scale and nature of the noise introduced by SGD and provides guidelines for the step size and batch size to use when training a neural network. We illustrate our findings on examples in which we train neural networks to learn the energy function of the continuous 3‐spin model on the sphere. The approximation error scales as our analysis predicts in as high a dimension as d = 25. © 2022 Courant Institute of Mathematics and Wiley Periodicals LLC.

Synchronization and pinning control of stochastic coevolving networks

Network dynamical systems are often characterized by the interlaced evolution of the node and edge dynamics, which are driven by both deterministic and stochastic factors. This manuscript offers a general mathematical model of coevolving network, which associates a state variable to each node and edge in the network, and describes their evolution through coupled stochastic differential equations. We study the emergence of synchronization, be it spontaneous or induced by a pinning control action, and provide sufficient conditions for local and global convergence. We enable the use of the Master Stability Function approach for studying coevolving networks, thereby obtaining conditions for almost sure local exponential convergence, whereas global conditions are derived using a Lyapunov-based approach. The theoretical results are then leveraged to design synchronization and pinning control protocols in two select applications. In the first one, the edge dynamics are tailored to induce spontaneous synchronization, whereas in the second the pinning edges are activated/deactivated and their weights modulated to drive the network towards the pinner’s trajectory in a distributed fashion.

A Mean-Field Analysis of a Network Behavioral–Epidemic Model

The spread of an epidemic disease and the population's collective behavioural response are deeply intertwined, influencing each other's evolution. Such a co-evolution typically has been overlooked in mathematical models, limiting their real-world applicability. To address this gap, we propose and analyse a behavioural-epidemic model, in which a susceptible-infected-susceptible epidemic model and an evolutionary game-theoretic decision-making mechanism concerning the use of self-protective measures are coupled. Through a mean-field approach, we characterise the asymptotic behaviour of the system, deriving conditions for global convergence to a disease-free equilibrium and characterising the endemic equilibria of the system and their (local) stability. Interestingly, for a certain range of the model parameters, we prove global convergence to a limit cycle, characterised by periodic epidemic outbreaks.

Good Characteristics of The New Spectral Conjugate Gradient Method for Unconstrained Optimization

The spectral conjugate gradient (SCG) method is an effective method to solve large-scale nonlinear unconstrained optimization problems. In this work, we propose a new SCG method in which performance is numerically analyzed. We established the descent property and global convergence conditions based on assumptions through the strongWolfe-Powell line search. Numerical results were performed using benchmark functions widely used in many conventional functions to evaluate the efficiency of the proposed method.Subject Classification: 90C30, 90C06, 65K05, 65K10.

A modified of FR method to solve unconstrained optimization

There are many methods derived from the conjugate gradient method, the most famous of which is the FR method (Fletcher–Reeves). Most of the methods are found to solve large unconstrained optimization problems. In this paper, we made a modified to the FR method, so that it achieves better numerical results as well as the conditions of global convergence. The numerical experiment showed the efficiency and robustness of the new method.

Proximal gradient flow and Douglas–Rachford splitting dynamics: Global exponential stability via integral quadratic constraints

Many large-scale and distributed optimization problems can be brought into a composite form in which the objective function is given by the sum of a smooth term and a nonsmooth regularizer. Such problems can be solved via a proximal gradient method and its variants, thereby generalizing gradient descent to a nonsmooth setup. In this paper, we view proximal algorithms as dynamical systems and leverage techniques from control theory to study their global properties. In particular, for problems with strongly convex objective functions, we utilize the theory of integral quadratic constraints to prove the global exponential stability of the equilibrium points of the differential equations that govern the evolution of proximal gradient and Douglas–Rachford splitting flows. In our analysis, we use the fact that these algorithms can be interpreted as variable-metric gradient methods on the suitable envelopes and exploit structural properties of the nonlinear terms that arise from the gradient of the smooth part of the objective function and the proximal operator associated with the nonsmooth regularizer. We also demonstrate that these envelopes can be obtained from the augmented Lagrangian associated with the original nonsmooth problem and establish conditions for global exponential convergence even in the absence of strong convexity.

Идентификация параметров дискретных стохастических систем методом обратных итераций

Получены условия глобальной сходимости алгоритмов, основанных на обратных итерациях в переменной метрике, в задаче идентификации параметров дискретной стохастической системы с возмущениями в невязке уравнения и наблюдениях процессов. Доказана сходимость оценок параметров к истинному значению при увеличении объема выборки наблюдений истинного процесса. Приведены примеры расчетов The article addresses the problem of identifying parameters of discrete stochastic systems with perturbations in the residual of the equation and observation of variables. The identification functional in the problem has a complex nature of isosurfaces, which is why universal minimization algorithms based on estimates of the first and second derivatives have a small radius of convergence. It is proposed to employ efficient computational identification algorithms with inverse iterations in a variable metric for solving the convergence problem for two classes of systems with simple correspondence between matrix elements and parameters of equivalent systems without state variables. These algorithms are used for systems without state variables due to the large radius and high convergence rate since the 1970s. At first, a theorem on the conditions for convergence of inverse iterations from almost any initial approximation to a small neighborhood of the global minimum of the identification functional was proved. Secondly, a theorem on the convergence of the points of the global minimum of the identification functional to the desired true value with an increase in the sample size of observations is taken into account. Assumption of a zero first and restricted second moments of stochastic disturbances in the residual of the equation and observation of variables was made. The convergence of inverse iterations is shown numerically in a model example with significant values of disturbances. The result of the article is new theorems on the conditions of global convergence of computational algorithms with inverse iterations in the problem with mixed disturbances and the justification of possibility of using these algorithms to identify the parameters for discrete stochastic systems of two classes with a simple correspondence between matrix elements and parameters of equivalent systems without state variables

New Parameter of CG-Method with Exact Line Search for Unconstraint Optimization

In this paper, a new CG method has been introduced to solve nonlinear equations systems. This method achieved the conditions of descent and global convergence, using the exact line search. The numerical results were good compared to other methods in terms of the number of iterations and the number of functions evaluation.

Improving Local Priority Hysteresis Switching Logic Convergence

Local priority hysteresis switching logic is associated with adaptive control convergence when using an infinite set of candidate parameters via constraints added to the switching scheme. In this paper, we reevaluate these constraints on the basis of the persistent excitation assumption. This makes room for the adaptive control to converge to its optimum, resulting in improved performance. Unconstrained local priority hysteresis switching logic is investigated, and global convergence conditions are proposed. This paper expands on the preliminary version of a conference paper [1] by adding numerical simulation examples to validate both the application and the advantage of the theory.

A two‐stage searching method for continuous time‐delay systems identification

AbstractThis paper presents a novel method to estimate the unknown parameters of continuous‐time systems with time delay. In the proposed method, the time delay and plant parameters are estimated separately. To estimate the time delay, a one‐dimensional searching method with variable step size is proposed to improve computational efficiency. The searching method consists of two stages: the coarse stage and the refined searching stage. To analyze the convergence of the searching method, the concept of significant interval is proposed. By defining the significant interval, a sufficient condition for global convergence of the searching method is provided. Based on the two‐stage searching method, a novel identification algorithm is developed in which the simplified refined instrumental variable for continuous‐time models algorithm is used to estimate the plant parameters. Simulation results demonstrate that the proposed identification method can estimate the unknown parameters of continuous‐time system with time delay efficiently. The estimation results under different noisy conditions verify the reliability and robustness of the proposed method. The applicability of the developed identification method is demonstrated by a practical example.

A modified modulus-based matrix splitting iteration method for solving implicit complementarity problems

In this paper, a modified modulus-based matrix splitting iteration method is established for solving a class of implicit complementarity problems. The global convergence conditions are given when the system matrix is a positive definite matrix or an H+-matrix, respectively. In addition, some numerical examples show that the proposed method is efficient.

Generalized convergence analysis of the fractional order systems

Abstract The aim of the present work is to generalize the contraction theory for the analysis of the convergence of fractional order systems for both continuous-time and discrete-time systems. Contraction theory is a methodology for assessing the stability of trajectories of a dynamical system with respect to one another. The result of this study is a generalization of the Lyapunov matrix equation and linear eigenvalue analysis. The proposed approach gives a necessary and sufficient condition for exponential and global convergence of nonlinear fractional order systems. The examples elucidate that the theory is very straightforward and exact.

Convergence in Networked Recursive Identification with Output Quantization

This paper analyzes conditions for global parametric convergence of a networked recursive identification algorithm. The FIR based algorithm accounts for networked delay and signal quantization. The paper constructs counterexamples to parametric convergence using low order dynamic models and an asymmetric binary quantizer. The associated ODEs of the first order model are analysed with analytical methods and by numerical simulation. In particular, the analysis proves that parametric convergence does not occur in case the input signal distribution is discrete taking a finite number of values, the problem is symmetric, or in case the input signal distribution, dynamic gain and switch point are such that there is no signal energy in an arbitrary small neighborhood of any switch point. Global parametric convergence is also proved for one of the low order cases.