Linear Function Approximation Research Articles

Adaptive Temporal Difference Learning With Linear Function Approximation.

This paper revisits the temporal difference (TD) learning algorithm for the policy evaluation tasks in reinforcement learning. Typically, the performance of TD(0) and TD( λ) is very sensitive to the choice of stepsizes. Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link between the TD(0) learning algorithm and the stochastic gradient methods, we develop a provably convergent adaptive projected variant of the TD(0) learning algorithm with linear function approximation that we term AdaTD(0). In contrast to the TD(0), AdaTD(0) is robust or less sensitive to the choice of stepsizes. Analytically, we establish that to reach an ϵ accuracy, the number of iterations needed is [Formula: see text] in the general case, where ρ represents the speed of the underlying Markov chain converges to the stationary distribution. This implies that the iteration complexity of AdaTD(0) is no worse than that of TD(0) in the worst case. When the stochastic semi-gradients are sparse, we provide theoretical acceleration of AdaTD(0). Going beyond TD(0), we develop an adaptive variant of TD( λ), which is referred to as AdaTD( λ). Empirically, we evaluate the performance of AdaTD(0) and AdaTD( λ) on several standard reinforcement learning tasks, which demonstrate the effectiveness of our new approaches.

Efficient compact linear programs for network revenue management

We are concerned with computing bid prices in network revenue management using approximate linear programming. It is well-known that affine value function approximations yield bid prices which are not sensitive to remaining capacity. The analytic reduction to compact linear programs allows the efficient computation of such bid prices. On the other hand, capacity-dependent bid prices can be obtained using separable piecewise linear value function approximations. Even though compact linear programs have been derived for this case also, they are still computationally much more expensive compared to using affine functions. We propose compact linear programs requiring substantially smaller computing times while, simultaneously, significantly improving the performance of capacity-independent bid prices. This simplification is achieved by taking into account remaining capacity only if it becomes scarce. Although our proposed linear programs are relaxations of the unreduced approximate linear programs, we conjecture equivalence and provide according numerical support. We measure the quality of an approximation by the difference between the expected performance of an induced policy and the corresponding theoretical upper bound. Using this paradigm in numerical experiments, we demonstrate the competitiveness of our proposed linear programs.

Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning

Motivated by applications in reinforcement learning (RL), we study a nonlinear stochastic approximation (SA) algorithm under Markovian noise, and establish its finite-sample convergence bounds under various stepsizes. Specifically, we show that when using constant stepsize (i.e., αk≡α), the algorithm achieves exponential fast convergence to a neighborhood (with radius O(αlog(1/α))) around the desired limit point. When using diminishing stepsizes with appropriate decay rate, the algorithm converges with rate O(log(k)/k). Our proof is based on Lyapunov drift arguments, and to handle the Markovian noise, we exploit the fast mixing of the underlying Markov chain. To demonstrate the generality of our theoretical results on Markovian SA, we use it to derive the finite-sample bounds of the popular Q-learning algorithm with linear function approximation, under a condition on the behavior policy. Importantly, we do not need to make the assumption that the samples are i.i.d., and do not require an artificial projection step in the algorithm. Numerical simulations corroborate our theoretical results.

Implementing an Approximation of Cumulative Prospect Theory into Mixed Linear Programming – an Application to Bio-Economic Modelling at Farm-Scale Considering Crop Insurance

Many empirical studies have found Cumulative Prospect Theory (CPT) superior in depicting risk behavior compared to the expected utility approach and literature now offers also CPT related parameter estimates for European farmers. CPT combines two segments of utility functions, a convex, risk loving one for losses and a concave, risk averse one for gains, and assigns subjective weights to the pay-offs according to their cumulative probabilities. So far, no implementation of CPT into constrained optimization problems exists, allowing for instance, the simulation of risk management under CPT in farm-scale programming models. To close this gap, we propose to combine endogenous sorting of the pay-offs based on integer variables with a piece-wise linear approximation of the value function using SOS2 (Special Ordered Sets of Type 2) variables. The SOS2 variables are required to deal with the convexity of the loss segment of the utility function. The integer sorting assigns the weights to the pay-offs according to their cumulative probabilities, it requires that all pay-offs are equally likely. Simulating optimal uptake levels of variants of a hypothetical crop insurance product with an evolved bio-economic model at farm-scale serves a proof of concept. The model considers adjustments in the cropping plan and allows for partial insurance coverage, in opposite to existing studies which evaluate the uptake of crop insurance at fixed crop choices and depict coverage as a yes-no decision. The approximation error of the approach is found as negligible small and the numerical burden compared to optimization under risk neutrality as still acceptable. The proposed approximation approach is quite general and applicable for any utility function increasing in the pay-off value and does not require its differentiability. It can also be applied without probability weighting. The empirical application underlines that the approach generates the expected behavior when a risk reducing strategy, here crop insurance, is considered under CPT. Insured acreage generally increases with higher strike levels where more frequently occurring but lower crop damages are covered, and with reduced cost of the insurance products. Using crop insurance as a risk management strategy is found to interact with other measures such as adjustments in cropping shares. This underlines the usefulness of an approach which allows to optimize interacting risk management strategies at farm-scale under CPT, considering resource and other relevant constraints.

Kutta–Joukowski Theorem for Unsteady Linear Aerodynamics

This paper presents the extension of the Kutta–Joukowski theorem to unsteady linear aerodynamics. Starting from the formulation developed by Theodorsen for the solution of the velocity potential for circulatory flows around thin, rectilinear airfoils, the frequency response function between bound circulation and circulatory lift is derived. To provide a formulation suitable for time-domain applications, two finite-state approximations of the Kutta–Joukowski frequency response function, with different degrees of complexity and accuracy, are also proposed. The validity of the derived unsteady Kutta–Joukowski theorem is verified by correlation with numerical predictions of the circulatory lift given by a validated boundary-element-method solver for potential flows. Furthermore, the proposed finite-state approximations of the unsteady Kutta–Joukowski theorem are applied to problems concerning the circulatory lift response to damped oscillatory airfoil motion and gust perturbation. The higher-accuracy finite-state model provides predictions that correspond perfectly to those obtained by the convolution integrals of the step responses given by the theories of Wagner and Küssner. The results given by the simpler finite-state model derived from the linear approximation of the frequency response function are satisfactory for low-frequency problems, and are compared with those provided by a widely used approximate unsteady version of the Kutta–Joukowski theorem available in the literature.

Using Reinforcement Learning to Control Traffic Signals in a Real-World Scenario: An Approach Based on Linear Function Approximation

Reinforcement learning is an efficient, widely used machine learning technique that performs well in problems with a reasonable number of states and actions. This is rarely the case regarding control-related problems, as for instance controlling traffic signals, where the state space can be very large. One way to deal with the curse of dimensionality is to use generalization techniques such as function approximation. In this paper, a linear function approximation is used by traffic signal agents in a network of signalized intersections. Specifically, a true online SARSA <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\lambda)$ </tex-math></inline-formula> algorithm with Fourier basis functions (TOS( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda $ </tex-math></inline-formula> )-FB) is employed. This method has the advantage of having convergence guarantees and error bounds, a drawback of non-linear function approximation. In order to evaluate TOS( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda $ </tex-math></inline-formula> )-FB, we perform experiments in variations of an isolated intersection scenario and a scenario of the city of Cottbus, Germany, with 22 signalized intersections, implemented in MATSim. We compare our results not only to fixed-time controllers, but also to a state-of-the-art rule-based adaptive method, showing that TOS( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda $ </tex-math></inline-formula> )-FB shows a performance that is highly superior to the fixed-time, while also being at least as efficient as the rule-based approach. For more than half of the intersections, our approach leads to less congestion and delay, without the need for the knowledge that underlies the rule-based approach.

DPCD: Discrete Principal Coordinate Descent for Binary Variable Problems

Binary optimization, a representative subclass of discrete optimization, plays an important role in mathematical optimization and has various applications in computer vision and machine learning. Generally speaking, binary optimization problems are NP-hard and difficult to solve due to the binary constraints, especially when the number of variables is very large. Existing methods often suffer from high computational costs or large accumulated quantization errors, or are only designed for specific tasks. In this paper, we propose an efficient algorithm, named Discrete Principal Coordinate Descent (DPCD), to find effective approximate solutions for general binary optimization problems. The proposed algorithm iteratively solves optimization problems related to the linear approximation of loss functions, which leads to updating the binary variables that most impact the value of the loss functions at each step. Our method supports a wide range of empirical objective functions with/without restrictions on the numbers of 1s and -1s in the binary variables. Furthermore, the theoretical convergence of our algorithm is proven, and the explicit convergence rates are derived for objective functions with Lipschitz continuous gradients, which are commonly adopted in practice. Extensive experiments on binary hashing tasks and large-scale datasets demonstrate the superiority of the proposed algorithm over several state-of-the-art methods in terms of both effectiveness and efficiency.

Differentially Private Reinforcement Learning with Linear Function Approximation

Motivated by the wide adoption of reinforcement learning (RL) in real-world personalized services, where users' sensitive and private information needs to be protected, we study regret minimization in finite-horizon Markov decision processes (MDPs) under the constraints of differential privacy (DP). Compared to existing private RL algorithms that work only on tabular finite-state, finite-actions MDPs, we take the first step towards privacy-preserving learning in MDPs with large state and action spaces. Specifically, we consider MDPs with linear function approximation (in particular linear mixture MDPs) under the notion of joint differential privacy (JDP), where the RL agent is responsible for protecting users' sensitive data. We design two private RL algorithms that are based on value iteration and policy optimization, respectively, and show that they enjoy sub-linear regret performance while guaranteeing privacy protection. Moreover, the regret bounds are independent of the number of states, and scale at most logarithmically with the number of actions, making the algorithms suitable for privacy protection in nowadays large-scale personalized services. Our results are achieved via a general procedure for learning in linear mixture MDPs under changing regularizers, which not only generalizes previous results for non-private learning, but also serves as a building block for general private reinforcement learning.

Multi-Agent Natural Actor-Critic Reinforcement Learning Algorithms

Multi-agent actor-critic algorithms are an important part of the Reinforcement Learning (RL) paradigm. We propose three fully decentralized multi-agent natural actor-critic (MAN) algorithms in this work. The objective is to collectively find a joint policy that maximizes the average long-term return of these agents. In the absence of a central controller and to preserve privacy, agents communicate some information to their neighbors via a time-varying communication network. We prove convergence of all the three MAN algorithms to a globally asymptotically stable set of the ODE corresponding to actor update; these use linear function approximations. We show that the Kullback–Leibler divergence between policies of successive iterates is proportional to the objective function’s gradient. We observe that the minimum singular value of the Fisher information matrix is well within the reciprocal of the policy parameter dimension. Using this, we theoretically show that the optimal value of the deterministic variant of the MAN algorithm at each iterate dominates that of the standard gradient-based multi-agent actor-critic (MAAC) algorithm. To our knowledge, it is the first such result in multi-agent reinforcement learning (MARL). To illustrate the usefulness of our proposed algorithms, we implement them on a bi-lane traffic network to reduce the average network congestion. We observe an almost 25% reduction in the average congestion in 2 MAN algorithms; the average congestion in another MAN algorithm is on par with the MAAC algorithm. We also consider a generic 15 agent MARL; the performance of the MAN algorithms is again as good as the MAAC algorithm.

Saturation problems for positive linear approximation of function in quasi normed spaces

Many authors work on constrained approximation such as monotonicity, convexity and k monotoni city, but little works introduced in positive collinear approximation. The aim of our paper is to investigate linear and positive approximation for real functions in , saturation problem between degree of best positive and linear approximation, and find collinear positive best approximation for measurable function in Lebesgue quasi normed spaces.

Linear Function Approximation as a Resource Efficient Method to Solve the Travelling Salesman Problem

This paper presents an approach to combinatorial optimization problems using linear function approximation (LFA) to solve the Travelling Salesman Problem (TSP). We create a reinforcement learning model in which we parameterize our policy using linear function approximation instead of the more commonly used neural networks. We then evaluated our models based on two factors: training time and optimality. When we compared our results with a state-of-the-art neural network solver, we found that our model was able to solve the TSP accurately while using drastically less computational resources and time to train than the neural network algorithm (Kool et al., 2019).

A CPU-FPGA Heterogeneous Acceleration System for Scene Text Detection Network

Scene text detection network, such as Connectionist Text Proposal Network (CTPN), takes CNN-RNN hybrid neural network as the main body, which can effectively recognize the text arranged horizontally in the image. However, complex structures and intensive calculations lead to longer network inference time. FPGAs have been widely used in data centers to accelerate neural network inference due to their flexibility and low power consumption. In this brief, a CPU-FPGA based heterogeneous acceleration system and a subgraph segmentation scheme for CNN-RNN hybrid neural networks are proposed. Winograd algorithm is applied to accelerate CNN. In RNN accelerating, fixed-point quantization, loop tiling, and piecewise linear approximation of activation function are used to reduce hardware resource usage and achieve a high degree of parallelization. CTPN is tested in our system with Intel Xeon 4116 CPU and Arria10 GX1150 FPGA, and the throughput reaches 1223.53GOP/s.

Dynamic Valuation of Battery Lifetime

This paper proposes a dynamic valuation framework to determine the opportunity value of battery capacity degradation in grid applications based on the internal degradation mechanism and utilization scenarios. The proposed framework follows a dynamic programming approach and includes a piecewise linear value function approximation solution that solves the optimization problem over a long planning horizon. The paper provides two case studies on price arbitrage and frequency regulation using real market and system data to demonstrate the broad applicability of the proposed framework. Results show that the battery lifetime value is critically dependent on both the external market environment and its internal state of health. On the grid service side, results show that second-life batteries can provide more than 50&#x0025; of the value compared to new batteries, and frequency regulation provides two times more revenue than price arbitrage throughout the battery lifetime.

An exact constant on the estimation of the approximation of classes of periodic functions of two variables by Ceśaro means

In the present work, we study problem related to the approximation of continuous $2\pi$-periodic functions by linear means of their Fourier series. The simplest example of a linear approximation of periodic function is the approximation of this function by partial sums of the Fourier series. However, as well known, the sequence of partial Fourier sums is not uniformly convergent over the class of continuous $2\pi$-periodic functions. Therefore, a significant number of papers is devoted to the research of the approximative properties of different approximation methods, which are generated by some transformations of the partial sums of the Fourier series. The methods allow us to construct sequence of trigonometrical polynomials that would be uniformly convergent for all functions $f \in C$. Particularly, Ceśaro means and Fejer sums have been widely studied in past decades.One of the important problems in this field is the study of the exact constant in an inequality for upper bounds of linear means deviations of the Fourier sums on fixed classes of periodic functions. Methods of investigation of integral representations for trigonometric polynomial deviations are generated by linear methods of summation of the Fourier series. They were developed in papers of Nikolsky, Stechkin, Nagy and others.&#x0D; The paper presents known results related to the approximation of classes of continuous functions by linear means of the Fourier sums and new facts obtained for some particular cases.In the paper, it is studied the approximation by the Ceśaro means of Fourier sums in Lipschitz class. In certain cases, the exact inequalities are found for upper bounds of deviations in the uniform metric of the second order rectangular Ceśaro means on the Lipschitz class of periodic functions in two variables.

Differentially Private Reinforcement Learning with Linear Function Approximation

Motivated by the wide adoption of reinforcement learning (RL) in real-world personalized services, where users' sensitive and private information needs to be protected, we study regret minimization in finite-horizon Markov decision processes (MDPs) under the constraints of differential privacy (DP). Compared to existing private RL algorithms that work only on tabular finite-state, finite-actions MDPs, we take the first step towards privacy-preserving learning in MDPs with large state and action spaces. Specifically, we consider MDPs with linear function approximation (in particular linear mixture MDPs) under the notion of joint differential privacy (JDP), where the RL agent is responsible for protecting users' sensitive data. We design two private RL algorithms that are based on value iteration and policy optimization, respectively, and show that they enjoy sub-linear regret performance while guaranteeing privacy protection. Moreover, the regret bounds are independent of the number of states, and scale at most logarithmically with the number of actions, making the algorithms suitable for privacy protection in nowadays large-scale personalized services. Our results are achieved via a general procedure for learning in linear mixture MDPs under changing regularizers, which not only generalizes previous results for non-private learning, but also serves as a building block for general private reinforcement learning.

Continuous Piecewise Linear Approximation of Plant-Based Hydro Production Function for Generation Scheduling Problems

An essential challenge in generation scheduling (GS) problems of hydrothermal power systems is the inclusion of adequate modeling of the hydroelectric production function (HPF). The HPF is a nonlinear and nonconvex function that depends on the head and turbined outflow. Although the hydropower plants have multiple generating units (GUs), due to a series of complexities, the most attractive modeling practice is to represent one HPF per plant, i.e., a single function is built for representing the plant generation instead of the generation of each GU. Furthermore, due to the computation time constraints and representation of nonlinearities, the HPF must be given by a piecewise linear (PWL) model. This paper presented some continuous PWL models to include the HPF per plant in GS problems of hydrothermal systems. Depending on the type of application, the framework allows a choice between the concave PWL for HPF modeled with one or two variables and the nonconvex (more accurate) PWL for HPF dependent only on the turbined outflow. Basically, in both PWL models, offline, mixed-integer linear (or quadratic) programming techniques are used with an optimized pre-selection of the original HPF dataset obtained through the Ramer-Douglas-Peucker algorithm. As a highlight, the framework allows the control of the number of hyperplanes and, consequently, the number of variables and constraints of the PWL model. To this end, we offer two possibilities: (i) minimizing the error for a fixed number of hyperplanes, or (ii) minimizing the number of hyperplanes for a given error. We assessed the performance of the proposed framework using data from two large hydropower plants of the Brazilian system. The first has 3568 MW distributed in 50 Bulb-type GUs and operates as a run-of-river hydro plant. In turn, the second, which can vary the reservoir volume by up to 1000 hm3, possesses 1140 MW distributed in three Francis-type units. The results showed a variation from 0.040% to 1.583% in terms of mean absolute error and 0.306% to 6.356% regarding the maximum absolute error even with few approximations.

Optimal Use and Replenishment of Two Substitutable Raw Materials in a Stochastic Capacitated Make-to-Order Production System

Problem definition: We study a multiperiod, nonstationary, make-to-order, joint production and inventory model where two kinds of input raw materials with availability uncertainties and different output conversion rates can be blended and then processed in a production line with stochastic capacity to produce the output product. Academic/practical relevance: The problem is motivated by the practice in coal-fired power plants, an important part of the energy sector, where two types of coal with different energy content per unit mass are blended for electrical power generation. Our model is the first to capture the key operational features in this context. Methodology: We model the problem as a Markov decision process and develop a novel approximate optimization approach to analyze and characterize the structure of the optimal policy. Results: We show that a use-down-to/balancing production policy and modified order-up-to ordering policy is optimal. We also propose a heuristic policy based on piece-wise linear value function approximation. Whereas computing the value function approximation via brute-force is time-consuming because of the curse of dimensionality, we leverage the structure of the optimal policy to develop an algorithm that greatly improves the computational time of the value function approximation. Our numerical studies on both a synthetic data set and real-world data show that the proposed heuristic provides significant profit improvement over three simpler straw policies, some of which are used in practice. Managerial implications: Our paper suggests the significant profit improvement opportunity of using our proposed policy and demonstrates how one can develop computationally more efficient heuristic policies by leveraging the structure of the optimal policy.

Structural and Statistical Analysis of Multidimensional Linear Approximations of Random Functions and Permutations

Structural and Statistical Analysis of Multidimensional Linear Approximations of Random Functions and Permutations

Finite-Sample Analysis of Off-Policy Natural Actor–Critic With Linear Function Approximation

In this paper, we develop a novel variant of natural actor-critic algorithm using off-policy sampling and linear function approximation, and establish a sample complexity of O(ε-3), outperforming all the previously known convergence bounds of such algorithms. In order to overcome the divergence due to deadly triad in off-policy policy evaluation under function approximation, we develop a critic that employs n-step TD-learning algorithm with a properly chosen n. We present finite-sample convergence bounds on this critic, which are of independent interest. Furthermore, we develop a variant of natural policy gradient under function approximation, with an improved convergence rate of O(1/T) after T iterations. Combining the finite sample bounds of the actor and the critic, we obtain an overall O(ε-3) sample complexity. Our results were derived solely based on the assumption that the behavior policy sufficiently explores the state-action space, which is a much lighter assumption compared to the related literature.

Momentary finite element for elasticity 3D problems

A description of a new 8-node finite element in the form of a hexahedron is given for solving elasticity 3D problems. This finite element has the following features. This is a linear approximation of functions in the element, one point of integration and taking into account the moments of forces in the element. The finite element is based on “rare mesh” FEM schemes—finite element schemes in the form of n-dimensional cubes (square, cube, etc.) with templates in the form of inscribed simplexes (triangle, tetrahedron, etc.). Among the rare mesh schemes, schemes in 3-dimensional and 7-dimensional spaces are successful, in which the simplex can be arranged symmetrically with respect to the center of the n-dimensional cube. The rare mesh FEM schemes have not the hourglass instability due to the fact that the template of the finite element operator has the form of a simplex. Compared to traditional linear finite elements in the form of a simplex, rare mesh schemes are more economical and converge better, since they do not have the effect of overestimated shear stiffness. Moment FEM schemes are constructed by rare mesh schemes higher dimensional projection, respectively, on a two-dimensional or three-dimensional finite element mesh. The resulting finite elements are close to the known polylinear elements and surpass them in efficiency. The schemes contain parameters that allow you to control the convergence of numerical solutions. The possibility of applying this approach to the construction of numerical schemes for solving other problems of mathematical physics is discussed.