Linear Function Approximation Research Articles

Whittle Index-Based Q-Learning for Wireless Edge Caching With Linear Function Approximation

Whittle Index-Based Q-Learning for Wireless Edge Caching With Linear Function Approximation

LinFa-Q: Accurate Q-learning with linear function approximation

Although Q-learning has achieved remarkable success in some practical cases, it often suffers from the overestimation problem in stochastic environments, which is commonly viewed as a shortcoming of Q-learning. Overestimated values are introduced by estimations on the next state Q value, which is well-known as the maximization bias. In this paper, we propose a more accurate method for estimating the Q value by the Q value decomposition and re-evaluation with similar samples based on linear function approximation. Specifically, we reform the parameterized incremental update formula of Q-learning and also demonstrate that the new formula is equivalent to the original one. Moreover, we propose a new parameterized incremental update formula of Q-learning to address the overestimation problem and present the more accurate computing method, which can be used in problems with continuous state spaces and stochastic environments. Experimentally, when compared with Doubly Bounded Q-learning and other Q-learning based methods, the new algorithm has more than 31% improvement of performance in Mountain Car and Cart Pole. Furthermore, the algorithm is robust to the learning rate and its memory capacity. Finally, the practical applicability of our algorithm is discussed through an analysis of time consumption.

REFINING THE EMPIRICAL MODEL OF THE HEAT TRANSFER BOUNDARY CONDITIONS OF THE PISTON OF A HIGH-SPEED DIESEL ENGINE

The level of temperature of ICE pistons is a factor that greatly affects their reliability. Therefore, the need to simulate the temperature state of pistons with sufficient accuracy and minimal allocation of resources and time exists in research and during design. Primary approaches for determining the boundary conditions of the problem of thermal conduction of the piston on the basis of criterion equations, high-level mathematical models, and empirical models are considered. The latter ones have become widespread by virtue of simplicity and ease of use in practice; their usability is limited to a certain set of operating modes of one engine or a few ones of a similar design. On the basis of data from four series of experimental tests for a well-studied diesel engine 4ChN12/14 the existing empirical models of boundary conditions, suitable for identifying the temperature state of the piston only within the respective separate series separately, are refined in the paper. Processing of the data set on the measured temperatures in the control zones of the piston was performed, and linear function approximations were found that satisfactorily describe the effect of diesel power on the specified temperatures. The model of boundary conditions for a set of eighteen separate regions on the surfaces of the piston was proposed, which takes into account the dependence of the piston temperature state on the brake power of the ICE, the crank angle of the start of fuel injection, and the conditions of cooling the piston with oil. The influence of the heat insulation layer on the piston crown surface on the heat transfer was accounted for by introducing an additional thermal resistance term into the boundary condition equations. The model made obtaining the temperature field of the piston possible in simulation that is consistent with the results of all series of experiments in the control zones of the piston periphery, the edge of the combustion chamber, the vicinity of the groove of the first piston ring, the cooled surface opposite to the combustion chamber, and the piston skirt. The importance of a gradual change in the temperature of the oil and coolant during the transition of the internal combustion engine from one mode of operation to another regarding the temperature state of the piston is demonstrated. Taking the temperatures of oil from previous modes of operation into account is recommended during the analysis of engine transient processes.

Decentralized Adaptive temporal-difference learning over time-varying networks and its finite-time analysis

In reinforcement learning, centralized temporal-difference (TD) learning is commonly used to solve the policy evaluation problem. However, the decentralized adaptive variant of the TD learning algorithm has rarely been investigated in multi-agent reinforcement learning. To fill this gap, based on linear function approximation, we propose a decentralized adaptive TD learning algorithm over time-varying networks, referred to as D-AdaTD. We rigorously analyze the convergence performance of D-AdaTD, i.e., the explicit finite-time analysis is established under different step-sizes. Specifically, under constant step-sizes, the average estimated value function can converge to a neighborhood of the optimal value at rate O(1/(k+1)) and the estimated parameter of each agent can converge to a neighborhood of the optimal parameter at rate O(ξk), where k is the number of iterations and ξ∈(0,1). Under diminishing step-sizes, the average estimated value function can converge to the optimal value and the average estimated parameter can converge to the optimal parameter at rate O((1+log(k+1))/k) and O((1+log(k+1))/(k+1)), respectively. In addition, we evaluate the performance of D-AdaTD via simulation experiments, which are commonly insufficient in the existing decentralized temporal-difference learning. The experimental results also validate the effectiveness of D-AdaTD.

High-Probability Sample Complexities for Policy Evaluation With Linear Function Approximation

High-Probability Sample Complexities for Policy Evaluation With Linear Function Approximation

Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

Optimal management of a microgrid Li-Ion battery considering non-linear losses using the Integer Zig-Zag formulation

This paper presents a model for optimizing the management of a Li-Ion battery in a microgrid, considering the presence of nonlinear losses during the charging and discharging processes. The paper examines the nature of these losses and proposes updated nonlinear expressions that depend on the state of charge and on the charging or discharging power. The piecewise linear approximations of the losses bivariate functions are implemented using the novel integer zig-zag (ZZI) formulation. The corresponding ZZI constraints are integrated into an optimization model to determine the optimal schedule for a residential isolated microgrid comprising a solar panel, a diesel generator, and a Li-Ion battery. The case study compares different triangulation strategies regarding their impact on the quality of the obtained solution and the computational burden.

Distributed entropy-regularized multi-agent reinforcement learning with policy consensus

Sample efficiency is a limiting factor for existing distributed multi-agent reinforcement learning (MARL) algorithms over networked multi-agent systems. In this paper, the sample efficiency problem is tackled by formally incorporating the entropy regularization into the distributed MARL algorithm design. Firstly, a new entropy-regularized MARL problem is formulated under the model of networked multi-agent Markov decision processes with observation-based policies and homogeneous agents, where the policy parameter sharing among the agents provably preserves the optimality. Secondly, an on-policy distributed actor–critic algorithm is proposed, where each agent shares its parameters of both the critic and actor for consensus update. Then, the convergence analysis of the proposed algorithm is provided based on the stochastic approximation theory under the assumption of linear function approximation of the critic. Furthermore, a practical off-policy version of the proposed algorithm is developed which possesses scalability, data efficiency and learning stability. Finally, the proposed distributed algorithm is compared against the solid baselines including two classic centralized training algorithms in the multi-agent particle environment, whose learning performance is empirically demonstrated through extensive simulation experiments.

Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration

This paper studies safe Reinforcement Learning (safe RL) with linear function approximation and under hard instantaneous constraints where unsafe actions must be avoided at each step. Existing studies have considered safe RL with hard instantaneous constraints, but their approaches rely on several key assumptions: (i) the RL agent knows a safe action set for every state or knows a safe graph in which all the state-action-state triples are safe, and (ii) the constraint/cost functions are linear. In this paper, we consider safe RL with instantaneous hard constraints without assumption (i) and generalize (ii) to Reproducing Kernel Hilbert Space (RKHS). Our proposed algorithm, LSVI-AE, achieves O(√{d³H⁴K}) regret and O(H √{dK}) hard constraint violation when the cost function is linear and O(H?ₖ √{K}) hard constraint violation when the cost function belongs to RKHS. Here K is the learning horizon, H is the length of each episode, and ?ₖ is the information gain w.r.t the kernel used to approximate cost functions. Our results achieve the optimal dependency on the learning horizon K, matching the lower bound we provide in this paper and demonstrating the efficiency of LSVI-AE. Notably, the design of our approach encourages aggressive policy exploration, providing a unique perspective on safe RL with general cost functions and no prior knowledge of safe actions, which may be of independent interest.

Dynamic Regret of Adversarial MDPs with Unknown Transition and Linear Function Approximation

We study reinforcement learning (RL) in episodic MDPs with adversarial full-information losses and the unknown transition. Instead of the classical static regret, we adopt dynamic regret as the performance measure which benchmarks the learner's performance with changing policies, making it more suitable for non-stationary environments. The primary challenge is to handle the uncertainties of unknown transition and unknown non-stationarity of environments simultaneously. We propose a general framework to decouple the two sources of uncertainties and show the dynamic regret bound naturally decomposes into two terms, one due to constructing confidence sets to handle the unknown transition and the other due to choosing sub-optimal policies under the unknown non-stationarity. To this end, we first employ the two-layer online ensemble structure to handle the adaptation error due to the unknown non-stationarity, which is model-agnostic. Subsequently, we instantiate the framework to three fundamental MDP models, including tabular MDPs, linear MDPs and linear mixture MDPs, and present corresponding approaches to control the exploration error due to the unknown transition. We provide dynamic regret guarantees respectively and show they are optimal in terms of the number of episodes K and the non-stationarity P̄ᴋ by establishing matching lower bounds. To the best of our knowledge, this is the first work that achieves the dynamic regret exhibiting optimal dependence on K and P̄ᴋ without prior knowledge about the non-stationarity for adversarial MDPs with unknown transition.

Advanced Optimal Control Problem and Numerical Method for its Solving

The advanced statement of the optimal control problem is presented. The difference between the extended setting of the problem and the classical one is that the model of the control object consists of two subsystems, a reference model, which generates an optimal motion path and a dynamic model of the control object with a system for stabilizing movement along the optimal trajectory. In the problem, it is necessary to find a program control function whose argument is time and a stabilization system function whose argument is the deviation of the state vector of the control object from the optimal program trajectory. The task has many initial conditions, one of which is used in the search for software control, and the rest for the search for a stabilization system. The control quality criterion is defined as the sum of the original quality criterion for all specified initial conditions. The procedure for trans forming the classical setting of the optimal control problem to an extended setting based on the refinement of the problem for its practical implementation is presented. To solve the extended optimal control problem, a universal numerical method is proposed based on a piecemeal linear approximation of the control function using evolutionary algorithms and symbol regression methods for structurally parametric optimization of the stabilization system function. An example of solving an extended optimal control problem for spatial motion by a quadcopter, which should conduct reconnaissance of a given territory in a minimum time, is given.

Asymptotic estimates for deviations of Fejér means on Poisson integrals

The work is devoted to the investigation of problem of approximation of continuous periodic functions of a real variable by trigonometric polynomials generated by linear methods of summation of Fourier series. The simplest example of the process of linear approximation of periodic functions is approximation of functions by partial sums of their Fourier series. However, sequences of partial Fourier sums are not uniformly convergent on the whole class of continuous periodic functions. Therefore, numerous studies are devoted to the study of approximation properties of approximating methods, which are generated by different transformations of sequences of partial sums of the Fourier series and allow obtaining sequences of trigonometric polynomials that are uniformly convergent for the entire class of continuous functions. In particular, Fejér means have been intensively studied in recent decades. One of the important tasks in this direction is study of the asymptotic behavior of the upper bounds of the deviations of trigonometric polynomials for a fixed class of periodic functions. The aim of the work is to systematize the known results concerning the approximation of classes of periodic functions of high smoothness by arithmetic means of Fourier sums, and to present new facts obtained for a more general case. In paper we investigate the asymptotic behavior of upper bounds on classes of Poisson integrals of periodic functions of real variable of deviations of linear means of Fourier series, which are constructed using the Fejér summation method. We study the classes consist of analytic functions of a real variable, which can be regularly extended into the corresponding strip of the complex plane. In the work, asymptotic inequalities for the upper bounds of the deviations of the Fejér means on the class of Poisson integrals were found. In certain case of parameters defining the class, the obtained formulas coincide with previously known asymptotic equalities.

Efficient continuous piecewise linear regression for linearising univariate non-linear functions

Due to their flexibility and ability to incorporate non-linear relationships, Mixed-Integer Non-Linear Programming (MINLP) approaches for optimization are commonly presented as a solution tool for real-world problems. Within this context, piecewise linear (PWL) approximations of non-linear continuous functions are useful, as opposed to non-linear machine learning-based approaches, since they enable the application of Mixed-Integer Linear Programming techniques in the MINLP framework, as well as retaining important features of the approximated non-linear functions, such as convexity. In this work, we extend upon fast algorithmic approaches for modeling discrete data using PWL regression by tuning them to allow the modeling of continuous functions. We show that if the input function is convex, then the convexity of the resulting PWL function is guaranteed. An analysis of the runtime of the presented algorithm shows which function characteristics affect the efficiency of the model, and which classes of functions can be modeled very quickly. Experimental results show that the presented approach is significantly faster than five existing approaches for modeling non-linear functions from the literature, at least 11 times faster on the tested functions, and up to a maximum speedup of more than 328,000. The presented approach also solves six benchmark problems for the first time.

Expected Policy Gradient for Network Aggregative Markov Games in Continuous Space.

In this article, we investigate the Nash-seeking problem of a set of agents, playing an infinite network aggregative Markov game. In particular, we focus on a noncooperative framework where each agent selfishly aims at maximizing its long-term average reward without having explicit information on the model of the environment dynamics and its own reward function. The main contribution of this article is to develop a continuous multiagent reinforcement learning (MARL) algorithm for the Nash-seeking problem in infinite dynamic games with convergence guarantee. To this end, we propose an actor-critic MARL algorithm based on expected policy gradient (EPG) with two general function approximators to estimate the value function and the Nash policy of the agents. We consider continuous state and action spaces and adopt a newly proposed EPG to alleviate the variance of the gradient approximation. Based on such formulation and under some conventional assumptions (e.g., using linear function approximators), we prove that the policies of the agents converge to the unique Nash equilibrium (NE) of the game. Furthermore, an estimation error analysis is conducted to investigate the effects of the error arising from function approximation. As a case study, the framework is applied on a cloud radio access network (C-RAN) by modeling the remote radio heads (RRHs) as the agents and the congestion of baseband units (BBUs) as the dynamics of the environment.

A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation

A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation

Off-Policy Prediction Learning: An Empirical Study of Online Algorithms.

Off-policy prediction-learning the value function for one policy from data generated while following another policy-is one of the most challenging problems in reinforcement learning. This article makes two main contributions: 1) it empirically studies 11 off-policy prediction learning algorithms with emphasis on their sensitivity to parameters, learning speed, and asymptotic error and 2) based on the empirical results, it proposes two step-size adaptation methods called and that help the algorithm with the lowest error from the experimental study learn faster. Many off-policy prediction learning algorithms have been proposed in the past decade, but it remains unclear which algorithms learn faster than others. In this article, we empirically compare 11 off-policy prediction learning algorithms with linear function approximation on three small tasks: the Collision task, the task, and the task. The Collision task is a small off-policy problem analogous to that of an autonomous car trying to predict whether it will collide with an obstacle. The and tasks are designed such that learning fast in them is challenging. In the Rooms task, the product of importance sampling ratios can be as large as 214 . To control the high variance caused by the product of the importance sampling ratios, step size should be set small, which, in turn, slows down learning. The task is more extreme in that the product of the ratios can become as large as 214 × 25 . The algorithms considered are Off-policy TD, five Gradient-TD algorithms, two Emphatic-TD algorithms, Vtrace, and variants of Tree Backup and ABQ that are applicable to the prediction setting. We found that the algorithms' performance is highly affected by the variance induced by the importance sampling ratios. Tree Backup, Vtrace, and ABTDare not affected by the high variance as much as other algorithms, but they restrict the effective bootstrapping parameter in a way that is too limiting for tasks where high variance is not present. We observed that Emphatic TDtends to have lower asymptotic error than other algorithms but might learn more slowly in some cases. Based on the empirical results, we propose two step-size adaptation algorithms, which we collectively refer to as the Ratchet algorithms, with the same underlying idea: keep the step-size parameter as large as possible and ratchet it down only when necessary to avoid overshoot. We show that the Ratchet algorithms are effective by comparing them with other popular step-size adaptation algorithms, such as the Adam optimizer.

Effective R-Matrix Parameterizations for Nuclear Data

We derive an effective Reich-Moore approximation (RMA) of the Wigner-EisenbudR-matrix formalism parameterized by complex-valued resonance energies and widths; this RMA exactly reproduces the total eliminated cross section. We show that resonance parameters evaluated for a conventional boundary conditions (BCs),Bc=Sc(E),are approximately equal to theR-matrix parameters in Park’s formalism by employing a linear approximation of the shift function therein [T.-S. Park, Phys. Rev. C106(2021) 064612]. We outline a method for converting Park’s observed reduced width amplitudes (RWAs) and their covariance matrix into Brune’s alternativeR-matrix RWAs and their covariance matrix [C. Brune, Phys. Rev. C66(2002) 044611]. We extend the Park’sR-matrix formalism into the complex plane by introducing a complex-valued basis set of eigenfunctions of a complex-symmetric (non-Hermitian) Hamiltonian in theR-matrix interior. We observe that itsR-matrix resonance energies and widths are directly related to the poles and residues, respectively, of Hwang’s sum-over-poles representation of cross sections [R.N. Hwang, Nucl. Sci. Eng.96(1987) 192].

НЕЛИНЕЙНЫЙ АНАЛИЗ МАССИВНЫХ ЖЕЛЕЗОБЕТОННЫХ КОНСТРУКЦИЙ С УЧЕТОМ ТРЕЩИНООБРАЗОВАНИЯ

A finite element method, as well as the algorithm and the program for solid reinforced concrete structures analysis have been developed, taking into account plastic deformations of concrete. A modified Willam & Warnke failure criterion was used, supplemented by a flow criterion. Two models of volumetric deformation of concrete have been developed: an elastic model under brittle fracture and an ideal elastic-plastic model. An eight-node solid finite element with linear approximation of displacement functions, which implements the deformation models above mentioned, is constructed. This finite element is adapted to the PRINS computational software, and as part of this program it can be used for physically nonlinear analysis of building structures containing three-dimensional reinforced concrete elements. Modern building codes prescribe to carry out calculations of concrete and reinforced concrete structures in a nonlinear formulation, taking into account the real properties of concrete and reinforcement. To verify the developed finite element, a series of test calculations of a beam in the condition of pure bending was carried out. Comparison of the calculation results with experimental data confirmed the high accuracy and reliability of the results obtained.

Piecewise reconstruction of membership function approximation errors for Takagi–Sugeno fuzzy control

We propose a conservative relaxed control method to stabilize Takagi–Sugeno (TS) fuzzy systems. First, a conventional piecewise linear function approximation method is adopted to consider the membership functions’ information in the closed-loop system’s stability analysis. Subsequently, a model reconstruction method is proposed to reformulate linear approximation error terms in the TS fuzzy form, which can effectively introduce more information into stability conditions and achieve better robust control effects. Then, based on a fuzzy Lyapunov function approach, control design conditions are derived in terms of linear matrix inequalities, with less conservatism for related results in the literature. Finally, via a benchmark experiment, it is verified that the proposed method can show better robustness with fewer decision variables in robust convex conditions.

Distributed Neural Learning Algorithms for Multiagent Reinforcement Learning

In this paper, the fully distributed neural learning algorithms by neural network approximation for networked multi-agent reinforcement learning (NMARL) are studied. To tackle the convergence analysis of methods in NMARL with tremendous state-action space, most of the existing distributed algorithms are designed by linear function approximation, which however would fall into a situation of poor expression. To conquer such limitation, the distributed neural learning algorithms are developed by using a novel neural network approximation that bridges the theory and practice of deep multi-agent reinforcement learning (DMARL). Specifically, inspired by the overparametrization method for minimizing mean-squared projected bellman error (MSPBE), the distributed neural learning algorithms with population semigradients and stochastic semigradients are respectively proposed to solve the NMARL problem. Furthermore, the convergence of the proposed algorithms are strictly given by employing the overparametrization method to establish the approximate stationary point of MSPBE to characterizes the algorithms toward the global optimum. Finally, some numerical simulations demonstrate the effectiveness of the distributed neural learning algorithms.