Asymptotic Convergence Properties Research Articles

On the Quadratic Convergence of Newton’s Method for Mean Field Games with Non-separable Hamiltonian

AbstractWe analyze asymptotic convergence properties of Newton’s method for a class of evolutive Mean Field Games systems with non-separable Hamiltonian arising in mean field type models with congestion. We prove the well posedness of the Mean Field Game system with non-separable Hamiltonian and of the linear system giving the Newton iterations. Then, by forward induction and assuming that the initial guess is sufficiently close to the solution of problem, we show a quadratic rate of convergence for the approximation of the Mean Field Game system by Newton’s method. We also consider the case of a nonlocal coupling, but with separable Hamiltonian, and we show a similar rate of convergence.

Functional-Input Gaussian Processes with Applications to Inverse Scattering Problems

Surrogate modeling based on Gaussian processes (GPs) has received increasing attention in the analysis of complex problems in science and engineering. Despite extensive studies on GP modeling, the developments for functional inputs are scarce. Motivated by an inverse scattering problem in which functional inputs representing the support and material properties of the scatterer are involved in the partial differential equations, a new class of kernel functions for functional inputs is introduced for GPs. Based on the proposed GP models, the asymptotic convergence properties of the resulting mean squared prediction errors are derived and the finite sample performance is demonstrated by numerical examples. In the application to inverse scattering, a surrogate model is constructed with functional inputs, which is crucial to recover the reflective index of an inhomogeneous isotropic scattering region of interest for a given far-field pattern.

Enhanced Global Optimization With Parallel Global and Local Structures for Real-Time Control Systems

In practice, objective functions of real-time control systems can have multiple local minimums or can dramatically change over the function space, making them hard to optimize. To efficiently optimize such systems, in this paper, we develop a parallel global optimization framework that combines direct search methods with parallel Bayesian optimization. It consists of an iterative global and local search that searches broadly through the entire global space for promising regions and then efficiently exploits each local promising region. We prove the asymptotic convergence properties of the proposed framework and conduct several numerical experiments to illustrate its empirical performance. We also provide a real-time control problem to illustrate the efficiency of our proposed algorithm. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by a collision avoidance problem of vessels aided with onboard agent-based simulations. The simulation on one vessel can predict potential conflicts with other vessels on a pre-defined trajectory. In heavy congestion regions, the environment is highly dynamic and thus it is difficult to find a much safer alternative trajectory if collision is predicted on the current one. Moreover, for such real-time decisions, the control system should be quick in response to improve safety. The proposed metamodel based algorithm is designed for quick decision in such highly dynamic systems. The algorithm employs a decomposition of the response surface to better handle the multi-modal surface resulting from the highly dynamic environment. Specifically, it first looks at the large-scale trend globally (filter out the many local fluctuations that may otherwise trap the algorithm) to locate potential promising regions and then proceeds to this local regions for more detailed local search. To make quick decisions, it uses fast direct search algorithms in the local search phase and applies a parallel search scheme to enjoy the abundant computing power. Both the theoretical analysis and the simulation studies demonstrate that the proposed algorithm can provide better decisions quickly. We also note that this algorithm is not limited to real-time control or simulation-based system. In the case where each run of the experiment is expensive and the budget is limited for the final decision and when the response function is multi-modal, this algorithm can hopefully become a quite efficient and competitive approach. The multi-modal responses have broad applications in the area of control, planning and operations research, such as robot navigating and reinforcement learning.

Neurodynamic approaches for multi-agent distributed optimization

This paper considers a class of multi-agent distributed convex optimization with a common set of constraints and provides several continuous-time neurodynamic approaches. In problem transformation, l1 and l2 penalty methods are used respectively to cast the linear consensus constraint into the objective function, which avoids introducing auxiliary variables and only involves information exchange among primal variables in the process of solving the problem. For nonsmooth cost functions, two differential inclusions with projection operator are proposed. Without convexity of the differential inclusions, the asymptotic behavior and convergence properties are explored. For smooth cost functions, by harnessing the smoothness of l2 penalty function, finite- and fixed-time convergent algorithms are provided via a specifically designed average consensus estimator. Finally, several numerical examples in the multi-agent simulation environment are conducted to illustrate the effectiveness of the proposed neurodynamic approaches.

A generalized conditional gradient method for multiobjective composite optimization problems

This article deals with multiobjective composite optimization problems that consist of simultaneously minimizing several objective functions, each of which is composed of a combination of smooth and non-smooth functions. To tackle these problems, we propose a generalized version of the conditional gradient method, also known as Frank-Wolfe method. The method is analysed with three step size strategies, including Armijo-type, adaptive, and diminishing step sizes. We establish asymptotic convergence properties and iteration-complexity bounds, with and without convexity assumptions on the objective functions. Numerical experiments illustrating the practical behaviour of the methods are presented.

Deep reinforcement learning using least‐squares truncated temporal‐difference

AbstractPolicy evaluation (PE) is a critical sub‐problem in reinforcement learning, which estimates the value function for a given policy and can be used for policy improvement. However, there still exist some limitations in current PE methods, such as low sample efficiency and local convergence, especially on complex tasks. In this study, a novel PE algorithm called Least‐Squares Truncated Temporal‐Difference learning (LST2D) is proposed. In LST2D, an adaptive truncation mechanism is designed, which effectively takes advantage of the fast convergence property of Least‐Squares Temporal Difference learning and the asymptotic convergence property of Temporal Difference learning (TD). Then, two feature pre‐training methods are utilised to improve the approximation ability of LST2D. Furthermore, an Actor‐Critic algorithm based on LST2D and pre‐trained feature representations (ACLPF) is proposed, where LST2D is integrated into the critic network to improve learning‐prediction efficiency. Comprehensive simulation studies were conducted on four robotic tasks, and the corresponding results illustrate the effectiveness of LST2D. The proposed ACLPF algorithm outperformed DQN, ACER and PPO in terms of sample efficiency and stability, which demonstrated that LST2D can be applied to online learning control problems by incorporating it into the actor‐critic architecture.

Improved front steepest descent for multi-objective optimization

In this paper, we deal with the Front Steepest Descent algorithm for multi-objective optimization. We point out that the algorithm from the literature is often incapable, by design, of spanning large portions of the Pareto front. We thus introduce some modifications within the algorithm aimed to overcome this significant limitation. We prove that the asymptotic convergence properties of the algorithm are preserved and numerically show that the proposed method significantly outperforms the original one.

Efficient reinforcement learning with least-squares soft Bellman residual for robotic grasping

Grasping control of intelligent robots has to deal with the difficulties of model uncertainties and nonlinearities. In this paper, we propose the Kernel-based Least-Squares Soft Bellman residual Actor–Critic (KLSAC) algorithm for robotic grasping. In the proposed approach, a novel linear temporal-difference learning algorithm using the least-squares soft Bellman residual (LS2BR) method is designed for policy evaluation. In addition, KLSAC adopts a sparse-kernel feature representation method based on approximate linear dependency (ALD) analysis to construct features for continuous state–action space. Compared with typical deep reinforcement learning algorithms, KLSAC has two main advantages: firstly, the critic module has the capacity for rapid convergence by computing the fixed point of the linear soft Bellman equation via the least-squares optimization method. Secondly, the kernel-based features construction approach only requires predefining the basic kernel function and can improve the generalization ability of KLSAC. The simulation studies on robotic grasping control were conducted in the V-REP simulator. The results demonstrate that compared with other typical RL algorithms (e.g., SAC and BMPO), the proposed KLSAC algorithm can achieve better performance in terms of sample efficiency and asymptotic convergence property. Furthermore, experimental results on a real UR5 robot validated that KLSAC performed well in the real world.

Robust Output Regulation for Uncertain Linear Minimum Phase Systems under Unknown Control Direction⋆

This paper investigates the problem of disturbance rejection for SISO uncertain linear minimum phase systems perturbed by an unmeasurable external disturbance under the framework of robust output regulation. The model parameters of systems in question are largely uncertain, including the control direction. In addition, the external disturbance is unstructured but bounded. Towards this end, a novel Unknown Input Observer (UIO)-based regulator is developed to reject the external disturbance, and a switching mechanism with a monitor function is designed to handle the control direction uncertainty. Notable features are that the unstructured external disturbance can be directly estimated and completely rejected by a sliding mode-based observer, and this new scheme can be applied for systems with non-unity relative degree under unknown control direction. The boundedness of closed-loop system and its asymptotic convergence properties are rigorously proved, which is verified by a numerical example.

Asymptotic convergence of a distributed weighted least squares algorithm for networked systems with vector node variables

This paper studies the convergence properties of a recently proposed distributed algorithm for weighted least-squares (WLS) estimation in networked systems. This algorithm is suitable for large-scale networks with a vector parameter (variable) in each node of the network. By establishing the connection between this algorithm and the celebrated Gaussian Belief Propagation (BP) algorithm for statistical learning with scalar variables, asymptotic convergence of the algorithm is established under the assumption of generalised block diagonal dominance. This result generalises the known asymptotic convergence result of the Gaussian BP algorithm for networks with scalar variables. By extending the notion of diagonal dominance to block matrices, we are able to generalise the so-called walk-sum approach for convergence analysis of the Gaussian BP algorithm to this distributed WLS algorithm and show a similar asymptotic convergence property for networks with vector parameters. The significance of our work is that it gives theoretical guarantee for the distributed WLS algorithm for a new class of large-scale networked systems with vector parameters.

Asymptotically tight conic approximations for chance-constrained AC optimal power flow

The increasing penetration of renewable energy in power systems calls for secure and reliable system operations under significant uncertainty. To that end, the chance-constrained AC optimal power flow (CC-ACOPF) problem has been proposed. Most research in the literature of CC-ACOPF focuses on one-sided chance constraints; however, two-sided chance constraints (TCCs), albeit more complex, provide more accurate formulations as both upper and lower bounds of the chance constraints are enforced simultaneously. In this paper, we introduce a fully two-sided CC-ACOPF problem (TCC-ACOPF), in which the active/reactive generation, voltage, and power flow all remain within their upper/lower bounds simultaneously with a predefined probability. Instead of applying Bonferroni approximation or scenario-based approaches, we present an efficient second-order cone programming (SOCP) approximation of the TCCs under Gaussian Mixture (GM) distribution via a piecewise linear (PWL) approximation. Compared to the conventional normality assumption for forecast errors, the GM distribution adds an extra level of accuracy representing the uncertainties. Moreover, we show that our SOCP formulation has adjustable rates of accuracy and its optimal value enjoys asymptotic convergence properties. Furthermore, an algorithm is proposed to speed up the solution procedure by optimally selecting the PWL segments. Finally, we demonstrate the effectiveness of our proposed approaches with both real historical data and synthetic data on the IEEE 30-bus and 118-bus systems. We show that our formulations provide significantly more robust solutions (about 60% reduction in constraint violation) compared to other state-of-art ACOPF formulations.

ISS inequalities for vector versions of Halanay’s inequality and of the trajectory-based approach

Halanay’s inequality is a powerful and widely used tool to prove asymptotic convergence properties of functions that arise in the study of systems with delays or continuous-discrete features. In its standard form, it applies to scalar valued functions that satisfy decay conditions, with overshoots depending on suprema of the functions over suitable intervals. Then it provides an exponential decay estimate on the scalar functions. Here, we provide vector versions of Halanay’s inequality, and of the so-called trajectory based approach, both yielding input-to-state stability (or ISS) inequalities. Our proofs of the inequalities use the theory of positive systems. We apply our results to prove ISS for interval observers and other cases.

Two fast variance-reduced proximal gradient algorithms for SMVIPs-Stochastic Mixed Variational Inequality Problems with suitable applications to stochastic network games and traffic assignment problems

In this paper, we propose two proximal gradient algorithms with variance reduction for stochastic mixed variational inequality problems. One is a proximal extragradient algorithm and another is a proximal forward–backward–forward algorithm. Under the monotonicity assumption on the mapping F and other moderate conditions, we derive some asymptotic convergence properties and O(1/k) convergence rate in terms of the restricted gap function values for the proposed algorithms. Furthermore, under the bounded metric subregularity condition, we investigate the linear convergence rate and oracle complexity bounds for the proposed algorithms when the sample-size increases at a geometric rate. If the sample-size increases at a polynomial rate of ⌈k+1⌉−s with s>0, the mean-squared distance of the iterates to the solution set decays at a corresponding polynomial rate, while the iterations and oracle complexities to obtain an ε-solution are O(1/ε1/s) and O(1/ε1+1/s) respectively. Finally, some numerical experiments on stochastic network games and traffic assignment problems indicate that the proposed algorithms are efficient.

Data-Driven Input Reconstruction and Experimental Validation

This paper proposes a data-driven input reconstruction method from outputs (IRO) based on the Willems&#x2019; Fundamental Lemma. Given only output measurements, the unknown inputs estimated recursively by the IRO asymptotically converge to the true input without knowing the initial conditions. A recursive IRO and a moving-horizon IRO are developed based respectively on Lyapunov conditions and Luenberger-observer-type feedback, and their asymptotic convergence properties are studied. An experimental study is presented demonstrating the efficacy of the moving-horizon IRO for estimating the occupancy of a building on the EPFL campus via measured carbon dioxide levels.

Nonlinear integral control with event-triggered feedback: Unknown decay rates, zeno-freeness, and asymptotic convergence

This paper studies the event-triggered implementation of integral control laws for nonlinear uncertain systems. It is assumed that the plant already admits an integral control law such that the closed-loop system is input-to-state stable (ISS) with respect to the state sampling errors and admits an ISS-Lyapunov function. The unknown offset between the plant equilibrium and the set point means that one may not find a uniform lower bound for all the possible decay rates of the ISS-Lyapunov function, which however is usually required in the existing results. The design in this paper is based on a sequence of lower bound estimates of the decay rate, and the proposed dynamic event trigger guarantees that the intersampling intervals are lower bounded by a positive constant, and the asymptotic convergence property of integral control is retained even with event-triggered feedback. The theoretical result is physically validated by a pendulum control system.

Improved Penalty Function with Memory for Stochastically Constrained Optimization via Simulation

Penalty function with memory (PFM) in Park and Kim [2015] is proposed for discrete optimization via simulation problems with multiple stochastic constraints where performance measures of both an objective and constraints can be estimated only by stochastic simulation. The original PFM is shown to perform well, finding a true best feasible solution with a higher probability than other competitors even when constraints are tight or near-tight. However, PFM applies simple budget allocation rules (e.g., assigning an equal number of additional observations) to solutions sampled at each search iteration and uses a rather complicated penalty sequence with several user-specified parameters. In this article, we propose an improved version of PFM, namely IPFM, which can combine the PFM with any simulation budget allocation procedure that satisfies some conditions within a general DOvS framework. We present a version of a simulation budget allocation procedure useful for IPFM and introduce a new penalty sequence, namely PS 2 + , which is simpler than the original penalty sequence yet holds convergence properties within IPFM with better finite-sample performances. Asymptotic convergence properties of IPFM with PS 2 + are proved. Our numerical results show that the proposed method greatly improves both efficiency and accuracy compared to the original PFM.

Dynamic ordering decisions with approximate learning of supply yield uncertainty

We consider the real-life problem of a coach bus manufacturer located in Turkey, facing the problem of setting ordering quantities for a part procured from an unreliable supplier, where the number of items delivered is binomially distributed with an unknown yield parameter, p. We use the well-defined finite-horizon planning context with deterministic demand per period, purchasing, holding, and shortage costs to investigate the effectiveness of a fill-rate based approximate learning scheme in comparison to an exact Bayesian learning scheme, where observations on the supplier's delivery performance are used to update the assumed distribution of p. We formulate the exact optimal learning problem as a Bayes-adaptive Markov decision process and solve the corresponding finite horizon stochastic dynamic program to provide insights on the value of online learning in comparison to the unrealistic perfect information (PI) and no information (NT) benchmarks. We contrast the performance of the so-called Bayesian Updating (BU) policy to other practical approaches such as using an assumed/guessed value of p and implementing a constant safety stock. Noting the significant value of learning, we finally study the effectiveness of an approximate learning formulation that does not enjoy the asymptotic consistency and convergence properties but involves much lower computational burden, and demonstrate its confounding performance, at times beating the BU policy with exact Bayesian updates.

Security correction control of stochastic cyber–physical systems subject to false data injection attacks with heterogeneous effects

In this paper, the interconnected observer intervention-based security correction control idea is proposed for stochastic cyber–physical systems (CPSs) subjected to false data injection attacks (FDIAs). The FDIAs are injected into the controller-to-actuator channel by the adversary via wireless transmission. In particular, the FDIAs with heterogeneous effects are constructed, which consist of periodic attacks with unknown parameters and bias injection attacks with asymptotic convergence property. A novel interconnected adaptive observer structure is designed to online estimate the heterogeneous attack effects. The security correction control scheme with resilience is presented by integrating interconnected adaptive observer and robust technology. It is demonstrated that the impaired state signals can be corrected and desired security performance can be guaranteed for stochastic CPSs under FDIAs with heterogeneous effects. Finally, two simulation verifications, including a F-16 longitudinal dynamics system controlled by network, are established to verify the validity and feasibility for the presented strategy.

Online Modeling of the CNC Engraving System With Dead-Zone Input Nonlinearity

This article is concerned with modeling of a class of permanent-magnet-synchronous-motor-based drive systems with dead-zone input nonlinearity. The experimental testing method based on recursive least squares (RLS) is combined with the mechanism analysis method based on physical laws. With the structure of the analytical model first determined by the mechanism analysis method, an online RLS-based data-driven method is then exploited to obtain the unknown parameters of the analytical model. Since the unknown parameters are decoupled in the proposed method, all parameter estimates are available. An auxiliary model and a variable forgetting factor are constructed to ensure the high accuracy and the fast convergence of the proposed method, respectively. Besides, the asymptotic convergence property of the proposed method is analyzed, and the upper bound of the parameter estimation error is also presented. The effectiveness of the proposed method is verified by simulations and experiments on a self-designed computer numerical control engraving system.

Conditional gradient method for multiobjective optimization

We analyze the conditional gradient method, also known as Frank–Wolfe method, for constrained multiobjective optimization. The constraint set is assumed to be convex and compact, and the objectives functions are assumed to be continuously differentiable. The method is considered with different strategies for obtaining the step sizes. Asymptotic convergence properties and iteration-complexity bounds with and without convexity assumptions on the objective functions are stablished. Numerical experiments are provided to illustrate the effectiveness of the method and certify the obtained theoretical results.