Linear Function Approximation Research Articles

Approximate dynamic programming approach to efficient metro train timetabling and passenger flow control strategy with stop-skipping

Urban metro systems continuously face high travel demand during rush hours, which brings excessive energy waste and high risk to passengers. In order to alleviate passenger congestion, improve train service levels and reduce energy consumption, a nonlinear dynamic programming (DP) model of efficient metro train timetabling and passenger flow control strategy with stop-skipping is presented, which consists of state transition equations concerning train traffic and passenger load. To overcome the curse of dimensionality, the formulated nonlinear DP problem is transformed into a discrete Markov decision process, and a novel approximate dynamic programming (ADP) approach is designed based on the lookahead policy and linear parametric value function approximation. Finally, the effectiveness of this method is verified by three groups of numerical experiments. Compared with Particle Swarm Optimization (PSO) and Simulated Annealing (SA), the designed ADP approach could obtain high-quality solutions quickly, which makes it applicable to the practical implementation of metro operations.

Distributed Actor-Critic Algorithms for Multiagent Reinforcement Learning Over Directed Graphs.

Actor-critic (AC) cooperative multiagent reinforcement learning (MARL) over directed graphs is studied in this article. The goal of the agents in MARL is to maximize the globally averaged return in a distributed way, i.e., each agent can only exchange information with its neighboring agents. AC methods proposed in the literature require the communication graphs to be undirected and the weight matrices to be doubly stochastic (more precisely, the weight matrices are row stochastic and their expectation are column stochastic). Differently from these methods, we propose a distributed AC algorithm for MARL over directed graph with fixed topology that only requires the weight matrix to be row stochastic. Then, we also study the MARL over directed graphs (possibly not connected) with changing topologies, proposing a different distributed AC algorithm based on the push-sum protocol that only requires the weight matrices to be column stochastic. Convergence of the proposed algorithms is proven for linear function approximation of the action value function. Simulations are presented to demonstrate the effectiveness of the proposed algorithms.

A risk-targeted decision model for the verification of the seismic performance of RC structures against structural pounding

In this study, a novel risk-targeted decision model is introduced to verify the seismic performance of RC structures against structural pounding and then to define the minimum demand for separation gap distance (dg) between adjacent structures at a target probability Pt. The derivations are based on a linear approximation of the seismic hazard function and well-established probabilistic seismic demand models (PSDMs). New performance objectives (POs) dedicated to structural pounding risk are proposed based on engineering demand parameter (EDP) hazard curves and on the separation gap distances (dg: available or acceptable) between adjacent structures. The theoretical background in the context of the risk-targeted decision model is presented, and then the adjustment to the structural pounding risk is introduced. The minimum separation gap distance (dg,minPt) at a target Pt is defined considering a) the demand hazard curve for the maximum displacement (without pounding) at the top contact floor level of the structure (δmax), and b) the acceptable structural pounding PO of an EDP. The proposed risk-targeted decision model is then demonstrated by means of a structural pounding case study of a RC structure. Local and global EDPs at different performance levels are considered and several POs at different dg are compounded. The verification of the seismic performance of the RC structure at a target probability Pt is carried out either with or without considering the structural pounding risk. Numerical examples of the verification and the definition of the dg,minPt at different pounding risk levels (Pt) of the examined RC structure are also presented and discussed. Based on these results, a distinct PO of an EDP was identified. This PO is characterized as a critical level of structural pounding risk at the capacity level of the EDP and is associated with an explicit-critical separation gap distance between the adjacent structures. In a future research study, this PO could be the basis for providing a design solution against structural pounding risk when the estimated minimum separation gap distance at a target Pt between the adjacent structures cannot be implemented. The provided solution through the proposed decision model also indicates that the maximum demand for separation gap distance between the adjacent structures can safely be shifted to smaller values, notwithstanding that pounding risk is increased.

Chemical space navigation by machine learning models for discovering selective MAO-B enzyme inhibitors for Parkinson’s disease

Monoamine Oxidase-B (MAO-B) is a key neuroprotective target that breaks neurotransmitters such as dopamine and releases highly reactive free radicals as the by-product. Its over-expression in the brain observed due to ageing and neurodegenerative diseases contributes to worsening neuronal degeneration. Being the primary enzyme for dopamine metabolism in the substantia nigra of the brain and due to the lack of efficient drug candidates, MAO-B selective, reversible inhibition is hot topic of research in Parkinson’s disease (PD). This study developed machine learning (ML) models that predict the activity of experimentally tested indole and indazole derivatives against MAO-B using linear genetic function approximation (GFA) and two non-linear support vector machine (SVM) and artificial neural network (ANN) techniques. ANN model with an R2 of 0.9704 for the training dataset, q2of 0.9436 for cross-validation and r2of 0.9025 for the test dataset were identified as the best-performing ML model with the seven significant molecular descriptors CATS2D_04_DA, CATS2D_05_DA, CATS3D_06_LL, Mor04u, Mor25m, P_VSA_v_2 and nO. The robust ML model was then employed to design novel MAO-B inhibitors with similar core scaffolds and their biological activity prediction. ANN model was further employed in the virtual screening of 4356 molecules from the ChEMBL database. Applicability domain analysis and pharmacokinetic and toxicity profiles predicted three newly designed molecules (22 N, 23 N and 24 N) and two virtually screened best ChEMBL molecules as potential drug candidates using the ANN ML model. Molecular docking studies of the best-identified compounds were performed to understand the molecular mechanism of interactions having high binding energy and selectivity with the MAO-B enzyme. The current study shortlisted 5 potential lead compounds as potent and selective MAO-B inhibitors, which could further be carried forward for in vitro and in vivo studies to discover small molecules against neurodegenerative disease.

A linear programming approach to difference-of-convex piecewise linear approximation

We address the problem of finding continuous piecewise linear (CPWL) approximations of deterministic functions of any dimension that satisfy any predefined error-tolerance, while keeping the number of polytopes that partition the approximation domain low. Specifically, we focus on overcoming the major computational bottleneck of the CPWL Approximation Algorithm (CPWL-AA) that has been proposed in the recent literature. CPWL-AA uses the difference-of-convex CPWL representation to search CPWL approximations which can partition the approximation domain to have polytopes of any shape. A computational bottleneck of the method is to solve a mixed-integer linear program (MILP) in which the number of binary variables is large for many problems of practical interest. In this paper, we overcome this by introducing a method that obtains a high quality solution of the MILP by iteratively solving a linear program (LP). We further reduce the computational expense by developing a method that treats some constraints in the LP problem as lazy constraints. Through a computational study we demonstrate that the proposed methods substantially reduce the computation time of CPWL-AA, while maintaining high quality CPWL approximations. With this, we demonstrate that we can generate CPWL approximations that satisfy predefined error-tolerances on functions of up to five dimensions within reasonable solution times.

On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation

Sample-efficient offline reinforcement learning (RL) with linear function approximation has been studied extensively recently. Much of the prior work has yielded instance-independent rates that hold even for the worst-case realization of problem instances. This work seeks to understand instance-dependent bounds for offline RL with linear function approximation. We present an algorithm called Bootstrapped and Constrained Pessimistic Value Iteration (BCP-VI), which leverages data bootstrapping and constrained optimization on top of pessimism. We show that under a partial data coverage assumption, that of concentrability with respect to an optimal policy, the proposed algorithm yields a fast rate for offline RL when there is a positive gap in the optimal Q-value functions, even if the offline data were collected adaptively. Moreover, when the linear features of the optimal actions in the states reachable by an optimal policy span those reachable by the behavior policy and the optimal actions are unique, offline RL achieves absolute zero sub-optimality error when the number of episodes exceeds a (finite) instance-dependent threshold. To the best of our knowledge, these are the first results that give a fast rate bound on the sub-optimality and an absolute zero sub-optimality bound for offline RL with linear function approximation from adaptive data with partial coverage. We also provide instance-agnostic and instance-dependent information-theoretical lower bounds to complement our upper bounds.

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

Approximation of classes of Poisson integrals by Fejer means

The paper is devoted to the investigation of problem of approximation of continuous periodic functions by trigonometric polynomials, which are generated by linear methods of summation of Fourier series.
 The simplest example of a linear approximation of periodic functions is the approximation of functions by partial sums of their Fourier series. However, the sequences of partial Fourier sums are not uniformly convergent over the class of continuous periodic functions. Therefore, many studies devoted to the research of the approximative properties of approximation methods, which are generated by transformations of the partial sums of Fourier series and allow us to construct sequences of trigonometrical polynomials that would be uniformly convergent for the whole class of continuous functions. Particularly, Fejer sums have been widely studied recently. One of the important problems in this area is the study of asymptotic behavior of the sharp upper bounds over a given class of functions of deviations of the trigonometric polynomials.
 In the paper, we study upper asymptotic estimates for deviations between a function and the Fejer means for the Fourier series of the function. The asymptotic behavior is considered for the functions represented by the Poisson integrals of periodic functions of a real variable. The mentioned classes consist of analytic functions of a real variable. These functions can be regularly extended into the corresponding strip of the complex plane.An asymptotic equality for the upper bounds of Fejer means deviations on classes of Poisson integrals was obtained.

A majorization–minimization-based method for nonconvex inverse rig problems in facial animation: algorithm derivation

Automated methods for facial animation are a necessary tool in the modern industry since the standard blendshape head models consist of hundreds of controllers, and a manual approach is painfully slow. Different solutions have been proposed that produce output in real-time or generalize well for different face topologies. However, all these prior works consider a linear approximation of the blendshape function and hence do not provide a high-enough level of detail for modern realistic human face reconstruction. A second-order blendshape approximation leads to higher fidelity facial animation but generates a non-linear least squares optimization problem with high dimensionality. We derive a method for solving the inverse rig in blendshape animation using quadratic corrective terms, which increases accuracy. At the same time, due to the proposed construction of the objective function, it yields a sparser estimated weight vector compared to the state-of-the-art methods. The former feature means lower demand for subsequent manual corrections of the solution, while the latter indicates that the manual modifications are also easier to include. Our algorithm is iterative and employs a Majorization–Minimization paradigm to cope with the increased complexity produced by adding corrective terms. The surrogate function is easy to solve and allows for further parallelization on the component level within each iteration. This paper is complementary to an accompanying paper (Racković et al. arxiv preprint. https://arxiv.org/abs/2302.04843, 2023) where we provide detailed experimental results and discussion, including highly-realistic animation data, and show a clear superiority of the results compared to the state-of-the-art methods.

Approximation of classes of Poisson integrals by Fejer means

The work is devoted to the investigation of problem of approximation of continuous periodic functions by trigonometric polynomials, which are generated by linear methods of summation of Fourier series. The simplest example of a linear approximation of periodic functions is the approximation of functions by partial sums of their Fourier series. However, the sequences of partial Fourier sums are not uniformly convergent over the class of continuous periodic functions. Therefore, a many studies is devoted to the research of the approximative properties of approximation methods, which are generated by transformations of the partial sums of Fourier series and allow us to construct sequences of trigonometrical polynomials that would be uniformly convergent for the whole class of continuous functions. Particularly, Fejer means have been widely studied in the last time. One of the important problems in this field is the study of asymptotic behavior of the upper bounds over a fixed classes of functions of deviations of the trigonometric polynomials. The aim of the work systematizes known results related to the approximation of classes of Poisson integrals of continuous functions by arithmetic means of Fourier sums, and presents new facts obtained for particular cases. The asymptotic behavior of the upper bounds on classes of Poisson integrals of periodic functions of the real variable of deviations of linear means of Fourier series, which are defined by applying the Fejer summation method is studied. The mentioned classes consist of analytic functions of a real variable, which are narrowing of bounded harmonic in unit disc functions of complex variable. In the work, asymptotic equalities for the upper bounds of deviations of Fejer means on classes of Poisson integrals were obtained.

Quantum Lazy Training

In the training of over-parameterized model functions via gradient descent, sometimes the parameters do not change significantly and remain close to their initial values. This phenomenon is called lazy training and motivates consideration of the linear approximation of the model function around the initial parameters. In the lazy regime, this linear approximation imitates the behavior of the parameterized function whose associated kernel, called the tangent kernel, specifies the training performance of the model. Lazy training is known to occur in the case of (classical) neural networks with large widths. In this paper, we show that the training of geometrically local parameterized quantum circuits enters the lazy regime for large numbers of qubits. More precisely, we prove bounds on the rate of changes of the parameters of such a geometrically local parameterized quantum circuit in the training process, and on the precision of the linear approximation of the associated quantum model function; both of these bounds tend to zero as the number of qubits grows. We support our analytic results with numerical simulations.

Provably Efficient Reinforcement Learning with Linear Function Approximation

Modern reinforcement learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy. The introduction of function approximation raises a fundamental set of challenges involving computational and statistical efficiency, especially given the need to manage the exploration/exploitation trade-off. As a result, a core RL question remains open: how can we design provably efficient RL algorithms that incorporate function approximation? This question persists even in a basic setting with linear dynamics and linear rewards, for which only linear function approximation is needed. This paper presents the first provable RL algorithm with both polynomial run time and polynomial sample complexity in this linear setting, without requiring a “simulator” or additional assumptions. Concretely, we prove that an optimistic modification of least-squares value iteration—a classical algorithm frequently studied in the linear setting—achieves [Formula: see text] regret, where d is the ambient dimension of feature space, H is the length of each episode, and T is the total number of steps. Importantly, such regret is independent of the number of states and actions. Funding: This work was supported by the Defense Advanced Research Projects Agency program on Lifelong Learning Machines.

Accelerated and Instance-Optimal Policy Evaluation with Linear Function Approximation

We study the problem of policy evaluation with linear function approximation and present efficient and practical algorithms that come with strong optimality guarantees. We begin by proving lower bounds that establish baselines on both the deterministic error and stochastic error in this problem. In particular, we prove an oracle complexity lower bound on the deterministic error in an instance-dependent norm associated with the stationary distribution of the transition kernel, and we use the local asymptotic minimax machinery to prove an instance-dependent lower bound on the stochastic error in the independent and identically distributed (i.i.d.) observation model. Existing algorithms fail to match at least one of these lower bounds: To illustrate, we analyze a variance-reduced variant of temporal difference learning, showing in particular that it fails to achieve the oracle complexity lower bound. To remedy this issue, we develop an accelerated, variance-reduced fast temporal difference (VRFTD) algorithm that simultaneously matches both lower bounds and attains a strong notion of instance-optimality. Finally, we extend the VRFTD algorithm to the setting with Markovian observations and provide instance-dependent convergence results. Our theoretical guarantees of optimality are corroborated by numerical experiments.

Distributed consensus-based multi-agent temporal-difference learning

In this paper we propose two new distributed consensus-based algorithms for temporal-difference learning in multi-agent Markov decision processes. The algorithms are of off-policy type and are aimed at linear approximation of the value function. Restricting agents’ observations to local data and communications to their small neighborhoods, the algorithms consist of: (a) local updates of the parameter estimates based on either the standard TD(λ) or the emphatic ETD(λ) algorithm, and (b) dynamic consensus scheme implemented over a time-varying lossy communication network. The algorithms are completely decentralized, allowing efficient parallelization and applications where the agents may have different behavior policies and different initial state distributions while evaluating a common target policy. It is proved under nonrestrictive assumptions that the proposed algorithms weakly converge to the solutions of the mean ordinary differential equation (ODE) common for all the agents. It is also proved that the whole system may be stabilized by a proper choice of the network and that the parameter estimates weakly converge to consensus. Discussion is given on the asymptotic bias and variance of the estimates, on the projected forms of the proposed algorithms, as well as on restrictiveness of the adopted assumptions. Simulation results illustrate the main properties of the algorithms and provide comparisons with similar schemes.

On the sample complexity of actor-critic method for reinforcement learning with function approximation

Reinforcement learning, mathematically described by Markov Decision Problems, may be approached either through dynamic programming or policy search. Actor-critic algorithms combine the merits of both approaches by alternating between steps to estimate the value function and policy gradient updates. Due to the fact that the updates exhibit correlated noise and biased gradient updates, only the asymptotic behavior of actor-critic is known by connecting its behavior to dynamical systems. This work puts forth a new variant of actor-critic that employs Monte Carlo rollouts during the policy search updates, which results in controllable bias that depends on the number of critic evaluations. As a result, we are able to provide for the first time the convergence rate of actor-critic algorithms when the policy search step employs policy gradient, agnostic to the choice of policy evaluation technique. In particular, we establish conditions under which the sample complexity is comparable to stochastic gradient method for non-convex problems or slower as a result of the critic estimation error, which is the main complexity bottleneck. These results hold in continuous state and action spaces with linear function approximation for the value function. We then specialize these conceptual results to the case where the critic is estimated by Temporal Difference, Gradient Temporal Difference, and Accelerated Gradient Temporal Difference. These learning rates are then corroborated on a navigation problem involving an obstacle and the pendulum problem which provide insight into the interplay between optimization and generalization in reinforcement learning.

Distributed TD(0) With Almost No Communication

We provide a new non-asymptotic analysis of distributed temporal difference learning with linear function approximation. Our approach relies on “one-shot averaging”, where N agents run identical local copies of the TD(0) method and average the outcomes only once at the very end. We demonstrate a version of the linear time speedup phenomenon, where the convergence time of the distributed process is a factor of N faster than the convergence time of TD(0). This is the first result proving benefits from parallelism for temporal difference methods.

Actor–Critic or Critic–Actor? A Tale of Two Time Scales

We revisit the standard formulation of tabular actor-critic algorithm as a two time-scale stochastic approximation with value function computed on a faster time-scale and policy computed on a slower time-scale. This emulates policy iteration. We observe that reversal of the time scales will in fact emulate value iteration and is a legitimate algorithm. We provide a proof of convergence and compare the two empirically with and without function approximation (with both linear and nonlinear function approximators) and observe that our proposed critic-actor algorithm performs on par with actor-critic in terms of both accuracy and computational effort.

L-SecNet: Towards Secure and Lightweight Deep Neural Network Inference

The advances in machine learning technology has promoted its great potential for deep neural network (DNN) inference powered applications of Internet of Things (IoT), such as facial verification cameras and speech recognition assistants. The current deployment of these applications also raises serious privacy concerns, especially when sensitive individual information is accessed easily by various IoT devices. Fortunately, the cryptography-based solutions are able to execute secure inference without infringing the user’s raw data and the model owner’s proprietary model. However, existing works suffer from impractically high latency and low accuracy, stemming primarily from the evaluations of the non-linear layers in DNN. In this paper, we propose L-SecNet, a lightweight secure neural network inference system that provides efficient inference services without sacrificing accuracy and privacy. Specifically, to reduce latency caused by comparison operations in non-linear layers, we subtly combine additive secret sharing and multiplicative secret sharing to design a lightweight secure comparison protocol. Further, we approximate the commonly used and time-consuming activation functions (including Sigmoid and Tanh functions) with the non-linear sin function instead of the linear polynomial approximation functions in the existing works. In order to maintain low running latency while meeting the requirement of high accuracy, a secure and lightweight protocol for a sin function is proposed. Our theoretical analysis and empirical experiments evaluate the security and efficiency of the L-SecNet system. Compared with the state-of-the-art works, L-SecNet saves up to about 80 times bandwidth and about 53 times runtime.

Optimal Oracle Inequalities for Projected Fixed-Point Equations, with Applications to Policy Evaluation

Linear fixed-point equations in Hilbert spaces arise in a variety of settings, including reinforcement learning, and computational methods for solving differential and integral equations. We study methods that use a collection of random observations to compute approximate solutions by searching over a known low-dimensional subspace of the Hilbert space. First, we prove an instance-dependent upper bound on the mean-squared error for a linear stochastic approximation scheme that exploits Polyak–Ruppert averaging. This bound consists of two terms: an approximation error term with an instance-dependent approximation factor and a statistical error term that captures the instance-specific complexity of the noise when projected onto the low-dimensional subspace. Using information-theoretic methods, we also establish lower bounds showing that both of these terms cannot be improved, again in an instance-dependent sense. A concrete consequence of our characterization is that the optimal approximation factor in this problem can be much larger than a universal constant. We show how our results precisely characterize the error of a class of temporal difference learning methods for the policy evaluation problem with linear function approximation, establishing their optimality. Funding: This work was partially supported by grants from the Office of Naval Research [Grant DOD-ONRN00014-18-1-2640] and the National Science Foundation (NSF) [NSF-IIS Grant 1909365, NSF-DMS Grant 2015454, and NSF-CCF Grant 1955450] to M. J. Wainwright. Part of this work was performed when A. Pananjady was visiting the Simons Institute for the Theory of Computing, where he was supported by a Swiss Re Research Fellowship. Supplemental Material: The online supplementary file is available at https://doi.org/10.1287/moor.2022.1341 .

Actuator fault reconstruction using FDI system based on sliding mode observers

Interplanetary space missions require spacecraft autonomy in order to fulfill the mission objective. The fault detection and isolation (FDI) system increases the level of autonomy and can ensure the safety of the spacecraft by detecting and isolating potential faults before they become critical. The proposed FDI system is based on an innovative bank of SMOs (sliding mode observers), designed for different fault scenarios cases. The FDI system design aims to detect and isolate actuators and measurement units’ faults used by the satellite control system and considers the nonlinear model of the satellite dynamics. This approach gives the possibility of fault reconstruction based on the information provided by an equivalent injection signal, allowing to reconstruct external perturbances and faults. The SMO chattering phenomenon is avoided by using the pseudo-sliding function, being a linear approximation of the signum function, which gives the possibility of using the equivalent injection signal for fault reconstruction purposes. The proposed fault reconstruction methodology is illustrated by a case study for a 6U Cubesat.