Linear Function Approximation Research Articles

High-Probability Sample Complexities for Policy Evaluation With Linear Function Approximation

High-Probability Sample Complexities for Policy Evaluation With Linear Function Approximation

Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

Distributed entropy-regularized multi-agent reinforcement learning with policy consensus

Sample efficiency is a limiting factor for existing distributed multi-agent reinforcement learning (MARL) algorithms over networked multi-agent systems. In this paper, the sample efficiency problem is tackled by formally incorporating the entropy regularization into the distributed MARL algorithm design. Firstly, a new entropy-regularized MARL problem is formulated under the model of networked multi-agent Markov decision processes with observation-based policies and homogeneous agents, where the policy parameter sharing among the agents provably preserves the optimality. Secondly, an on-policy distributed actor–critic algorithm is proposed, where each agent shares its parameters of both the critic and actor for consensus update. Then, the convergence analysis of the proposed algorithm is provided based on the stochastic approximation theory under the assumption of linear function approximation of the critic. Furthermore, a practical off-policy version of the proposed algorithm is developed which possesses scalability, data efficiency and learning stability. Finally, the proposed distributed algorithm is compared against the solid baselines including two classic centralized training algorithms in the multi-agent particle environment, whose learning performance is empirically demonstrated through extensive simulation experiments.

Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration

This paper studies safe Reinforcement Learning (safe RL) with linear function approximation and under hard instantaneous constraints where unsafe actions must be avoided at each step. Existing studies have considered safe RL with hard instantaneous constraints, but their approaches rely on several key assumptions: (i) the RL agent knows a safe action set for every state or knows a safe graph in which all the state-action-state triples are safe, and (ii) the constraint/cost functions are linear. In this paper, we consider safe RL with instantaneous hard constraints without assumption (i) and generalize (ii) to Reproducing Kernel Hilbert Space (RKHS). Our proposed algorithm, LSVI-AE, achieves O(√{d³H⁴K}) regret and O(H √{dK}) hard constraint violation when the cost function is linear and O(H?ₖ √{K}) hard constraint violation when the cost function belongs to RKHS. Here K is the learning horizon, H is the length of each episode, and ?ₖ is the information gain w.r.t the kernel used to approximate cost functions. Our results achieve the optimal dependency on the learning horizon K, matching the lower bound we provide in this paper and demonstrating the efficiency of LSVI-AE. Notably, the design of our approach encourages aggressive policy exploration, providing a unique perspective on safe RL with general cost functions and no prior knowledge of safe actions, which may be of independent interest.

Dynamic Regret of Adversarial MDPs with Unknown Transition and Linear Function Approximation

We study reinforcement learning (RL) in episodic MDPs with adversarial full-information losses and the unknown transition. Instead of the classical static regret, we adopt dynamic regret as the performance measure which benchmarks the learner's performance with changing policies, making it more suitable for non-stationary environments. The primary challenge is to handle the uncertainties of unknown transition and unknown non-stationarity of environments simultaneously. We propose a general framework to decouple the two sources of uncertainties and show the dynamic regret bound naturally decomposes into two terms, one due to constructing confidence sets to handle the unknown transition and the other due to choosing sub-optimal policies under the unknown non-stationarity. To this end, we first employ the two-layer online ensemble structure to handle the adaptation error due to the unknown non-stationarity, which is model-agnostic. Subsequently, we instantiate the framework to three fundamental MDP models, including tabular MDPs, linear MDPs and linear mixture MDPs, and present corresponding approaches to control the exploration error due to the unknown transition. We provide dynamic regret guarantees respectively and show they are optimal in terms of the number of episodes K and the non-stationarity P̄ᴋ by establishing matching lower bounds. To the best of our knowledge, this is the first work that achieves the dynamic regret exhibiting optimal dependence on K and P̄ᴋ without prior knowledge about the non-stationarity for adversarial MDPs with unknown transition.

Advanced Optimal Control Problem and Numerical Method for its Solving

The advanced statement of the optimal control problem is presented. The difference between the extended setting of the problem and the classical one is that the model of the control object consists of two subsystems, a reference model, which generates an optimal motion path and a dynamic model of the control object with a system for stabilizing movement along the optimal trajectory. In the problem, it is necessary to find a program control function whose argument is time and a stabilization system function whose argument is the deviation of the state vector of the control object from the optimal program trajectory. The task has many initial conditions, one of which is used in the search for software control, and the rest for the search for a stabilization system. The control quality criterion is defined as the sum of the original quality criterion for all specified initial conditions. The procedure for trans forming the classical setting of the optimal control problem to an extended setting based on the refinement of the problem for its practical implementation is presented. To solve the extended optimal control problem, a universal numerical method is proposed based on a piecemeal linear approximation of the control function using evolutionary algorithms and symbol regression methods for structurally parametric optimization of the stabilization system function. An example of solving an extended optimal control problem for spatial motion by a quadcopter, which should conduct reconnaissance of a given territory in a minimum time, is given.

Asymptotic estimates for deviations of Fejér means on Poisson integrals

The work is devoted to the investigation of problem of approximation of continuous periodic functions of a real variable by trigonometric polynomials generated by linear methods of summation of Fourier series. The simplest example of the process of linear approximation of periodic functions is approximation of functions by partial sums of their Fourier series. However, sequences of partial Fourier sums are not uniformly convergent on the whole class of continuous periodic functions. Therefore, numerous studies are devoted to the study of approximation properties of approximating methods, which are generated by different transformations of sequences of partial sums of the Fourier series and allow obtaining sequences of trigonometric polynomials that are uniformly convergent for the entire class of continuous functions. In particular, Fejér means have been intensively studied in recent decades. One of the important tasks in this direction is study of the asymptotic behavior of the upper bounds of the deviations of trigonometric polynomials for a fixed class of periodic functions. The aim of the work is to systematize the known results concerning the approximation of classes of periodic functions of high smoothness by arithmetic means of Fourier sums, and to present new facts obtained for a more general case. In paper we investigate the asymptotic behavior of upper bounds on classes of Poisson integrals of periodic functions of real variable of deviations of linear means of Fourier series, which are constructed using the Fejér summation method. We study the classes consist of analytic functions of a real variable, which can be regularly extended into the corresponding strip of the complex plane. In the work, asymptotic inequalities for the upper bounds of the deviations of the Fejér means on the class of Poisson integrals were found. In certain case of parameters defining the class, the obtained formulas coincide with previously known asymptotic equalities.

Efficient continuous piecewise linear regression for linearising univariate non-linear functions

Due to their flexibility and ability to incorporate non-linear relationships, Mixed-Integer Non-Linear Programming (MINLP) approaches for optimization are commonly presented as a solution tool for real-world problems. Within this context, piecewise linear (PWL) approximations of non-linear continuous functions are useful, as opposed to non-linear machine learning-based approaches, since they enable the application of Mixed-Integer Linear Programming techniques in the MINLP framework, as well as retaining important features of the approximated non-linear functions, such as convexity. In this work, we extend upon fast algorithmic approaches for modeling discrete data using PWL regression by tuning them to allow the modeling of continuous functions. We show that if the input function is convex, then the convexity of the resulting PWL function is guaranteed. An analysis of the runtime of the presented algorithm shows which function characteristics affect the efficiency of the model, and which classes of functions can be modeled very quickly. Experimental results show that the presented approach is significantly faster than five existing approaches for modeling non-linear functions from the literature, at least 11 times faster on the tested functions, and up to a maximum speedup of more than 328,000. The presented approach also solves six benchmark problems for the first time.

Expected Policy Gradient for Network Aggregative Markov Games in Continuous Space.

In this article, we investigate the Nash-seeking problem of a set of agents, playing an infinite network aggregative Markov game. In particular, we focus on a noncooperative framework where each agent selfishly aims at maximizing its long-term average reward without having explicit information on the model of the environment dynamics and its own reward function. The main contribution of this article is to develop a continuous multiagent reinforcement learning (MARL) algorithm for the Nash-seeking problem in infinite dynamic games with convergence guarantee. To this end, we propose an actor-critic MARL algorithm based on expected policy gradient (EPG) with two general function approximators to estimate the value function and the Nash policy of the agents. We consider continuous state and action spaces and adopt a newly proposed EPG to alleviate the variance of the gradient approximation. Based on such formulation and under some conventional assumptions (e.g., using linear function approximators), we prove that the policies of the agents converge to the unique Nash equilibrium (NE) of the game. Furthermore, an estimation error analysis is conducted to investigate the effects of the error arising from function approximation. As a case study, the framework is applied on a cloud radio access network (C-RAN) by modeling the remote radio heads (RRHs) as the agents and the congestion of baseband units (BBUs) as the dynamics of the environment.

НЕЛИНЕЙНЫЙ АНАЛИЗ МАССИВНЫХ ЖЕЛЕЗОБЕТОННЫХ КОНСТРУКЦИЙ С УЧЕТОМ ТРЕЩИНООБРАЗОВАНИЯ

A finite element method, as well as the algorithm and the program for solid reinforced concrete structures analysis have been developed, taking into account plastic deformations of concrete. A modified Willam & Warnke failure criterion was used, supplemented by a flow criterion. Two models of volumetric deformation of concrete have been developed: an elastic model under brittle fracture and an ideal elastic-plastic model. An eight-node solid finite element with linear approximation of displacement functions, which implements the deformation models above mentioned, is constructed. This finite element is adapted to the PRINS computational software, and as part of this program it can be used for physically nonlinear analysis of building structures containing three-dimensional reinforced concrete elements. Modern building codes prescribe to carry out calculations of concrete and reinforced concrete structures in a nonlinear formulation, taking into account the real properties of concrete and reinforcement. To verify the developed finite element, a series of test calculations of a beam in the condition of pure bending was carried out. Comparison of the calculation results with experimental data confirmed the high accuracy and reliability of the results obtained.

Piecewise reconstruction of membership function approximation errors for Takagi–Sugeno fuzzy control

We propose a conservative relaxed control method to stabilize Takagi–Sugeno (TS) fuzzy systems. First, a conventional piecewise linear function approximation method is adopted to consider the membership functions’ information in the closed-loop system’s stability analysis. Subsequently, a model reconstruction method is proposed to reformulate linear approximation error terms in the TS fuzzy form, which can effectively introduce more information into stability conditions and achieve better robust control effects. Then, based on a fuzzy Lyapunov function approach, control design conditions are derived in terms of linear matrix inequalities, with less conservatism for related results in the literature. Finally, via a benchmark experiment, it is verified that the proposed method can show better robustness with fewer decision variables in robust convex conditions.

Distributed Neural Learning Algorithms for Multiagent Reinforcement Learning

In this paper, the fully distributed neural learning algorithms by neural network approximation for networked multi-agent reinforcement learning (NMARL) are studied. To tackle the convergence analysis of methods in NMARL with tremendous state-action space, most of the existing distributed algorithms are designed by linear function approximation, which however would fall into a situation of poor expression. To conquer such limitation, the distributed neural learning algorithms are developed by using a novel neural network approximation that bridges the theory and practice of deep multi-agent reinforcement learning (DMARL). Specifically, inspired by the overparametrization method for minimizing mean-squared projected bellman error (MSPBE), the distributed neural learning algorithms with population semigradients and stochastic semigradients are respectively proposed to solve the NMARL problem. Furthermore, the convergence of the proposed algorithms are strictly given by employing the overparametrization method to establish the approximate stationary point of MSPBE to characterizes the algorithms toward the global optimum. Finally, some numerical simulations demonstrate the effectiveness of the distributed neural learning algorithms.

Approximate dynamic programming approach to efficient metro train timetabling and passenger flow control strategy with stop-skipping

Urban metro systems continuously face high travel demand during rush hours, which brings excessive energy waste and high risk to passengers. In order to alleviate passenger congestion, improve train service levels and reduce energy consumption, a nonlinear dynamic programming (DP) model of efficient metro train timetabling and passenger flow control strategy with stop-skipping is presented, which consists of state transition equations concerning train traffic and passenger load. To overcome the curse of dimensionality, the formulated nonlinear DP problem is transformed into a discrete Markov decision process, and a novel approximate dynamic programming (ADP) approach is designed based on the lookahead policy and linear parametric value function approximation. Finally, the effectiveness of this method is verified by three groups of numerical experiments. Compared with Particle Swarm Optimization (PSO) and Simulated Annealing (SA), the designed ADP approach could obtain high-quality solutions quickly, which makes it applicable to the practical implementation of metro operations.

Distributed Actor-Critic Algorithms for Multiagent Reinforcement Learning Over Directed Graphs.

Actor-critic (AC) cooperative multiagent reinforcement learning (MARL) over directed graphs is studied in this article. The goal of the agents in MARL is to maximize the globally averaged return in a distributed way, i.e., each agent can only exchange information with its neighboring agents. AC methods proposed in the literature require the communication graphs to be undirected and the weight matrices to be doubly stochastic (more precisely, the weight matrices are row stochastic and their expectation are column stochastic). Differently from these methods, we propose a distributed AC algorithm for MARL over directed graph with fixed topology that only requires the weight matrix to be row stochastic. Then, we also study the MARL over directed graphs (possibly not connected) with changing topologies, proposing a different distributed AC algorithm based on the push-sum protocol that only requires the weight matrices to be column stochastic. Convergence of the proposed algorithms is proven for linear function approximation of the action value function. Simulations are presented to demonstrate the effectiveness of the proposed algorithms.

A risk-targeted decision model for the verification of the seismic performance of RC structures against structural pounding

In this study, a novel risk-targeted decision model is introduced to verify the seismic performance of RC structures against structural pounding and then to define the minimum demand for separation gap distance (dg) between adjacent structures at a target probability Pt. The derivations are based on a linear approximation of the seismic hazard function and well-established probabilistic seismic demand models (PSDMs). New performance objectives (POs) dedicated to structural pounding risk are proposed based on engineering demand parameter (EDP) hazard curves and on the separation gap distances (dg: available or acceptable) between adjacent structures. The theoretical background in the context of the risk-targeted decision model is presented, and then the adjustment to the structural pounding risk is introduced. The minimum separation gap distance (dg,minPt) at a target Pt is defined considering a) the demand hazard curve for the maximum displacement (without pounding) at the top contact floor level of the structure (δmax), and b) the acceptable structural pounding PO of an EDP. The proposed risk-targeted decision model is then demonstrated by means of a structural pounding case study of a RC structure. Local and global EDPs at different performance levels are considered and several POs at different dg are compounded. The verification of the seismic performance of the RC structure at a target probability Pt is carried out either with or without considering the structural pounding risk. Numerical examples of the verification and the definition of the dg,minPt at different pounding risk levels (Pt) of the examined RC structure are also presented and discussed. Based on these results, a distinct PO of an EDP was identified. This PO is characterized as a critical level of structural pounding risk at the capacity level of the EDP and is associated with an explicit-critical separation gap distance between the adjacent structures. In a future research study, this PO could be the basis for providing a design solution against structural pounding risk when the estimated minimum separation gap distance at a target Pt between the adjacent structures cannot be implemented. The provided solution through the proposed decision model also indicates that the maximum demand for separation gap distance between the adjacent structures can safely be shifted to smaller values, notwithstanding that pounding risk is increased.

Chemical space navigation by machine learning models for discovering selective MAO-B enzyme inhibitors for Parkinson’s disease

Monoamine Oxidase-B (MAO-B) is a key neuroprotective target that breaks neurotransmitters such as dopamine and releases highly reactive free radicals as the by-product. Its over-expression in the brain observed due to ageing and neurodegenerative diseases contributes to worsening neuronal degeneration. Being the primary enzyme for dopamine metabolism in the substantia nigra of the brain and due to the lack of efficient drug candidates, MAO-B selective, reversible inhibition is hot topic of research in Parkinson’s disease (PD). This study developed machine learning (ML) models that predict the activity of experimentally tested indole and indazole derivatives against MAO-B using linear genetic function approximation (GFA) and two non-linear support vector machine (SVM) and artificial neural network (ANN) techniques. ANN model with an R2 of 0.9704 for the training dataset, q2of 0.9436 for cross-validation and r2of 0.9025 for the test dataset were identified as the best-performing ML model with the seven significant molecular descriptors CATS2D_04_DA, CATS2D_05_DA, CATS3D_06_LL, Mor04u, Mor25m, P_VSA_v_2 and nO. The robust ML model was then employed to design novel MAO-B inhibitors with similar core scaffolds and their biological activity prediction. ANN model was further employed in the virtual screening of 4356 molecules from the ChEMBL database. Applicability domain analysis and pharmacokinetic and toxicity profiles predicted three newly designed molecules (22 N, 23 N and 24 N) and two virtually screened best ChEMBL molecules as potential drug candidates using the ANN ML model. Molecular docking studies of the best-identified compounds were performed to understand the molecular mechanism of interactions having high binding energy and selectivity with the MAO-B enzyme. The current study shortlisted 5 potential lead compounds as potent and selective MAO-B inhibitors, which could further be carried forward for in vitro and in vivo studies to discover small molecules against neurodegenerative disease.

A linear programming approach to difference-of-convex piecewise linear approximation

We address the problem of finding continuous piecewise linear (CPWL) approximations of deterministic functions of any dimension that satisfy any predefined error-tolerance, while keeping the number of polytopes that partition the approximation domain low. Specifically, we focus on overcoming the major computational bottleneck of the CPWL Approximation Algorithm (CPWL-AA) that has been proposed in the recent literature. CPWL-AA uses the difference-of-convex CPWL representation to search CPWL approximations which can partition the approximation domain to have polytopes of any shape. A computational bottleneck of the method is to solve a mixed-integer linear program (MILP) in which the number of binary variables is large for many problems of practical interest. In this paper, we overcome this by introducing a method that obtains a high quality solution of the MILP by iteratively solving a linear program (LP). We further reduce the computational expense by developing a method that treats some constraints in the LP problem as lazy constraints. Through a computational study we demonstrate that the proposed methods substantially reduce the computation time of CPWL-AA, while maintaining high quality CPWL approximations. With this, we demonstrate that we can generate CPWL approximations that satisfy predefined error-tolerances on functions of up to five dimensions within reasonable solution times.

On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation

Sample-efficient offline reinforcement learning (RL) with linear function approximation has been studied extensively recently. Much of the prior work has yielded instance-independent rates that hold even for the worst-case realization of problem instances. This work seeks to understand instance-dependent bounds for offline RL with linear function approximation. We present an algorithm called Bootstrapped and Constrained Pessimistic Value Iteration (BCP-VI), which leverages data bootstrapping and constrained optimization on top of pessimism. We show that under a partial data coverage assumption, that of concentrability with respect to an optimal policy, the proposed algorithm yields a fast rate for offline RL when there is a positive gap in the optimal Q-value functions, even if the offline data were collected adaptively. Moreover, when the linear features of the optimal actions in the states reachable by an optimal policy span those reachable by the behavior policy and the optimal actions are unique, offline RL achieves absolute zero sub-optimality error when the number of episodes exceeds a (finite) instance-dependent threshold. To the best of our knowledge, these are the first results that give a fast rate bound on the sub-optimality and an absolute zero sub-optimality bound for offline RL with linear function approximation from adaptive data with partial coverage. We also provide instance-agnostic and instance-dependent information-theoretical lower bounds to complement our upper bounds.

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

Approximation of classes of Poisson integrals by Fejer means

The paper is devoted to the investigation of problem of approximation of continuous periodic functions by trigonometric polynomials, which are generated by linear methods of summation of Fourier series.
 The simplest example of a linear approximation of periodic functions is the approximation of functions by partial sums of their Fourier series. However, the sequences of partial Fourier sums are not uniformly convergent over the class of continuous periodic functions. Therefore, many studies devoted to the research of the approximative properties of approximation methods, which are generated by transformations of the partial sums of Fourier series and allow us to construct sequences of trigonometrical polynomials that would be uniformly convergent for the whole class of continuous functions. Particularly, Fejer sums have been widely studied recently. One of the important problems in this area is the study of asymptotic behavior of the sharp upper bounds over a given class of functions of deviations of the trigonometric polynomials.
 In the paper, we study upper asymptotic estimates for deviations between a function and the Fejer means for the Fourier series of the function. The asymptotic behavior is considered for the functions represented by the Poisson integrals of periodic functions of a real variable. The mentioned classes consist of analytic functions of a real variable. These functions can be regularly extended into the corresponding strip of the complex plane.An asymptotic equality for the upper bounds of Fejer means deviations on classes of Poisson integrals was obtained.