Temporal Difference Methods Research Articles

Reinforcement Learning-Based Security/Safety UAV System for Intrusion Detection Under Dynamic and Uncertain Target Movement

Autonomous security unmanned aerial vehicles (UAVs) have recently gained popularity as an effective solution for accomplishing target/intrusion detection and tracking tasks with little or no human intervention. In this context, we aim at developing an autonomous UAV system for detecting a dynamic and uncertain intrusion within an area, in which the intruder/target moves from one location to another within the area according to an unknown random distribution. The problem of finding an uncertain target while considering the energy causality constraint of the UAV&#x2019;s battery and the uncertainty of the target movement is mathematically formulated as a search benefit maximization problem, which cannot be directly optimized due to the uncertain unknown target movement. Thus, we reformulate the optimization problem as a Markov-decision process that can be solved using reinforcement learning (RL) techniques. Then, we implement an RL-based algorithm to solve the reformulated benefit maximization problem by enabling the UAV to autonomously learn the dynamics of the intruder/target. Specifically, different design variants of the RL-based algorithm are implemented that differ in the used temporal difference methods (i.e., Q-learning or state-action-reward-state-action), and in the exploration algorithms (convergence-based or <formula><tex>$\epsilon$</tex></formula>-greedy). Simulation results show RL algorithms&#x2019; superiority and effectiveness over existing random and circular target detection algorithms.

Kernel-Based Decentralized Policy Evaluation for Reinforcement Learning.

We investigate the decentralized nonparametric policy evaluation problem within reinforcement learning (RL), focusing on scenarios where multiple agents collaborate to learn the state-value function using sampled state transitions and privately observed rewards. Our approach centers on a regression-based multistage iteration technique employing infinite-dimensional gradient descent (GD) within a reproducing kernel Hilbert space (RKHS). To make computation and communication more feasible, we employ Nyström approximation to project this space into a finite-dimensional one. We establish statistical error bounds to describe the convergence of value function estimation, marking the first instance of such analysis within a fully decentralized nonparametric framework. We compare the regression-based method to the kernel temporal difference (TD) method in some numerical studies.

Boosting On-Policy Actor-Critic With Shallow Updates in Critic.

Deep reinforcement learning (DRL) benefits from the representation power of deep neural networks (NNs), to approximate the value function and policy in the learning process. Batch reinforcement learning (BRL) benefits from stable training and data efficiency with fixed representation and enjoys solid theoretical analysis. This work proposes least-squares deep policy gradient (LSDPG), a hybrid approach that combines least-squares reinforcement learning (RL) with online DRL to achieve the best of both worlds. LSDPG leverages a shared network to share useful features between policy (actor) and value function (critic). LSDPG learns policy, value function, and representation separately. First, LSDPG views deep NNs of the critic as a linear combination of representation weighted by the weights of the last layer and performs policy evaluation with regularized least-squares temporal difference (LSTD) methods. Second, arbitrary policy gradient algorithms can be applied to improve the policy. Third, an auxiliary task is used to periodically distill the features from the critic into the representation. Unlike most DRL methods, where the critic algorithms are often used in a nonstationary situation, i.e., the policy to be evaluated is changing, the critic in LSDPG is working on a stationary case in each iteration of the critic update. We prove that, under some conditions, the critic converges to the regularized TD fixpoint of current policy, and the actor converges to the local optimal policy. The experimental results on challenging Procgen benchmark illustrate the improvement of sample efficiency of LSDPG over proximal policy optimization and phasic policy gradient (PPG).

Time two-grid technique combined with temporal second order difference method for semilinear fractional diffusion-wave equations

Time two-grid technique combined with temporal second order difference method for semilinear fractional diffusion-wave equations

Monte Carlo and Temporal Difference Methods in Reinforcement Learning [AI-eXplained

Monte Carlo and Temporal Difference Methods in Reinforcement Learning [AI-eXplained

Formal Specification and Testing for Reinforcement Learning

The development process for reinforcement learning applications is still exploratory rather than systematic. This exploratory nature reduces reuse of specifications between applications and increases the chances of introducing programming errors. This paper takes a step towards systematizing the development of reinforcement learning applications. We introduce a formal specification of reinforcement learning problems and algorithms, with a particular focus on temporal difference methods and their definitions in backup diagrams. We further develop a test harness for a large class of reinforcement learning applications based on temporal difference learning, including SARSA and Q-learning. The entire development is rooted in functional programming methods; starting with pure specifications and denotational semantics, ending with property-based testing and using compositional interpreters for a domain-specific term language as a test oracle for concrete implementations. We demonstrate the usefulness of this testing method on a number of examples, and evaluate with mutation testing. We show that our test suite is effective in killing mutants (90% mutants killed for 75% of subject agents). More importantly, almost half of all mutants are killed by generic write-once-use-everywhere tests that apply to any reinforcement learning problem modeled using our library, without any additional effort from the programmer.

Optimal consensus control for multi‐agent systems: Multi‐step policy gradient adaptive dynamic programming method

AbstractThis paper presents a novel adaptive dynamic programming (ADP) method to solve the optimal consensus problem for a class of discrete‐time multi‐agent systems with completely unknown dynamics. Different from the classical RL‐based optimal control algorithms based on one‐step temporal difference method, a multi‐step‐based (also call n‐step) policy gradient ADP (MS‐PGADP) algorithm, which have been proved to be more efficient owing to its faster propagation of the reward, is proposed to obtain the iterative control policies. Moreover, a novel Q‐function is defined, which estimates the performance of performing an action in the current state. Then, through the Lyapunov stability theorem and functional analysis, the proof of optimality of the performance index function is given and the stability of the error system is also proved. Furthermore, the actor‐critic neural networks are used to implement the proposed method. Inspired by deep Q network, the target network is also introduced to guarantee the stability of NNs in the process of training. Finally, two simulations are conducted to verify the effectiveness of the proposed algorithm.

An efficient algorithm for the numerical solution of telegraph interface model with discontinuous coefficients via Haar wavelets

In this article, a hybrid numerical method based on Haar wavelets and finite difference method is proposed for the solution of hyperbolic telegraph interface model in one spacial dimension. We considered the problems having both constant and variable coefficients around interfaces of discontinuity across a fixed interface. In this method, the highest order spatial derivative is approximated by truncated Haar series, while for temporal derivative finite difference method is utilized. The proposed method is applied to some benchmark linear and nonlinear test problems. The exact and approximate solutions are compared for different number of collocation points at different time steps. The algorithm based on this new method is simple and can be easily implemented in any programming language. Experimental rates of convergence of the proposed method are calculated which are in agreement with theoretical results. The proposed method perform very well and produces a stable solution if sharp transitions exists in the solution space or if there is a discontinuity between initial and boundary conditions, whereas, the other existing method loses its accuracy in such cases. The numerical experiments, stability and rate of convergence both theoretical and computational, confirm the accuracy and diverse applicability of the method.

Online maximization of extracted energy in sea wave energy converters using temporal-difference learning

This article presents temporal-difference (TD) learning which is a combination of Monte Carlo and dynamic programing (DP) as a Method for controlling single-body wave energy converters (WECs). Since TD methods are designed to solve the prediction problems, we use this feature to maximize the energy captured from the sea waves. The entered force to the buoy system is addressed implicitly in the state matrix to design the problem into a TD framework. In order to enhance the captured power by the WEC, the control method is built to have an online active control. This will help the device to predict the best controller based on its previous experiences in the same situations. Two methods of TD, Q-Learning and SARSA, are used and the features are analyzed and several testing functions are carried out in simulation part. To perform on-line optimal control, a force control has acted as a controller and TD coefficients are tuned at a proper rate significantly after specific number of episodes. The power of suggested TD methods is compared to PGM, IPOPT and with other learning control strategies. Several computer simulations were carried out to evaluate the controller effectiveness by applying different sea-states and analyzing the resultant WEC dynamics.

Distributed TD(0) With Almost No Communication

We provide a new non-asymptotic analysis of distributed temporal difference learning with linear function approximation. Our approach relies on “one-shot averaging”, where N agents run identical local copies of the TD(0) method and average the outcomes only once at the very end. We demonstrate a version of the linear time speedup phenomenon, where the convergence time of the distributed process is a factor of N faster than the convergence time of TD(0). This is the first result proving benefits from parallelism for temporal difference methods.

Learning attack-defense response in continuous-time discrete-states Stackelberg Security Markov games

ABSTRACT Researchers have become interested in security games in recent decades as a result of its successful application in real-world security issues. The security model is based on the Stackelberg Security Game (SSG), in which defenders (leaders) select a defensive strategy based on the optimal reaction of attackers (followers), who, at equilibrium, select the predicted assaulting strategy as a response. These applications, on the other hand, do not account for the time constraints posed by the game’s players’ journey time. Furthermore, players should be able to cope with dynamic settings in which their knowledge of the environment changes on a regular basis, allowing them to perform more effectively. This research proposes a security model based on a continuous-time Reinforcement Learning (RL) approach implemented using a temporal difference method that takes prior information into account to address these issues. We use a controlled, ergodic continuous-time Markov game to model the SSG. The game framework model assumes that all information is available. We calculate the number of transitions over a time interval divided by the entire value of the holding time to estimate the transition rates. The arithmetic mean of the observed cost of the individual players is used to estimate the cost for defenders and attackers. An iterated proximal/gradient approach is used to calculate the SSG equilibrium point. We offer a continuous-time random walk method for game implementation. In a numerical case relevant to rain-forest hazards, we analyze the performance of the suggested RL security solution and discuss the problems that should be considered in future.

A prototype biospecklemeter for microbiological analysis: a starting point for a potential digital-image laser antibiotic susceptibility test

This work describes the design and construction of a low-cost prototype biospecklemeter. In this first report of its applications a microbiological analysis was performed. The relationship between microorganism density, motility and biospeckle activity was evaluated. The aim is to produce a device for research and for student use. The biospecklemeter was constructed on an antivibration table with low-cost materials and accessories and assayed using different microorganisms grown in Mueller–Hinton broth and/or agar as well as urine samples. Three lasers were installed (blue, green and red) as well as two webcams and a charge-coupled device camera. The samples were carried within the wells of a VDRL plate and 5–30 s videos or photographs of reflected light were taken. Video or photo processing was carried out using the method of temporal differences (ImageJ) or generalized differences (ImageDP). It was possible to establish a direct relationship between spectrophotometric absorption, the microbial count and the biospeckle intensity; changes in motility and the effect of antibiotics on bacterial laboratory samples as well as on urine samples were also detected. Finally, with this method colonies of different microorganisms on Mueller–Hinton agar were studied.

Deep neural networks based temporal-difference methods for high-dimensional parabolic partial differential equations

Solving high-dimensional partial differential equations (PDEs) is a long-standing challenge, for which classical numerical methods suffer from the well-known curse of dimensionality. In this paper, we propose deep neural networks (NN) based temporal-difference (TD) learning methods for numerically solving high-dimensional parabolic PDEs. To this end, we approximate the solution of the original PDE with an NN function. To calculate this neural network function, we first transform the deterministic parabolic PDE to a forward-backward stochastic differential equations (FBSDE) system with the nonlinear Feynman-Kac formula, and then transform the forward updating process of FBSDE as a Markov reward process (MRP). On this basis, we approximate the solution of the PDE by training an NN function with a reinforcement learning technique described as below: we first discretize the temporal interval to a finite number of time steps and then at each time step, we generate many trajectories and design the loss function as the mean square of temporal difference error on all trajectories. With this loss function, we update the parameters of the NN function with the stochastic gradient descent (SGD) method. Numerical experiments show that comparing to some other existed deep learning methods, our method not only accelerates the computational speed but also improves the computational accuracy. In particular, the relative errors of our algorithm achieve the order of O(10−4) even the dimension of the problem is as high as 100.

Multi-AUV Cooperative Navigation Algorithm Based on Temporal Difference Method

To reduce the cooperative positioning error and improve the navigation accuracy, a single master–slave AUV cooperative navigation method is proposed in this paper, which mainly focuses on planning the optimal path of the master AUV by the time difference (TD) method, under the premise that the path of the slave AUV has been planned. First, the model of multi-AUV cooperative navigation is established, and the observable problem of the system is analyzed. Second, for the single master–slave AUV cooperative navigation system, a Markov decision process (MDP)-based multi-AUV cooperative navigation model is established, and the master AUV path planning method is designed based on the TD method. Finally, the extended Kalman filter (EKF) and unscented Kalman filter (UKF) nonlinear filtering algorithms are applied to simulate and verify the algorithm that is proposed in this paper. The results show that the theoretical positioning error of the slave AUV can be controlled to about 3.2m by planning the path of the master AUV using the TD method. This method can not only reduce the observation error and positioning error of the slave AUV during the whole cooperative navigation process, but also keep the relative measurement distance between the master AUV and the slave AUV within an appropriate range.

A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions

Estimating value functions is a core component of reinforcement learning algorithms. Temporal difference (TD) learning algorithms use bootstrapping, i.e. they update the value function toward a learning target using value estimates at subsequent time-steps. Alternatively, the value function can be updated toward a learning target constructed by separately predicting successor features (SF)—a policy-dependent model—and linearly combining them with instantaneous rewards. We focus on bootstrapping targets used when estimating value functions, and propose a new backup target, the ?-return mixture, which implicitly combines value-predictive knowledge (used by TD methods) with (successor) feature-predictive knowledge—with a parameter ? capturing how much to rely on each. We illustrate that incorporating predictive knowledge through an ??-discounted SF model makes more efficient use of sampled experience, compared to either extreme, i.e. bootstrapping entirely on the value function estimate, or bootstrapping on the product of separately estimated successor features and instantaneous reward models. We empirically show this approach leads to faster policy evaluation and better control performance, for tabular and nonlinear function approximations, indicating scalability and generality.

Adaptive tracking control for stochastic nonlinear systems with time-varying delays using multi-dimensional Taylor network

For stochastic nonlinear systems (SNSs) perturbed by compound uncertainties, the conventional model-based control approaches assume that the evolution behavior of uncertain variables is known. Unfortunately, such approaches are often conservative for most practical scenarios with the slow convergence speed and unsatisfactory anti-interference performance. For this sake, an adaptive control scheme based on deep deterministic policy gradient (DDPG) and multi-dimensional Taylor network (MTN) is proposed here to address the tracking problem for a category of SNSs subject to fast time-varying uncertainties, stochastic disturbance and unknown time-varying delays. The effect of time delay is embedded in the reproducing kernel Hilbert space through the error coordinate transformation. In the framework of DDPG, the MTN-based surrogate is utilized to construct the online network and target network via the temporal-difference method, which promises more desirable real-time performance due to its concise structure than conventional NN-based surrogates. In order to enhance the robustness of the system under fast time-varying uncertainties, a novel persistent excitation (PE) mechanism is designed to ensure that the control policy is appropriately rewarded or punished. Based on the PE condition, weights of MTNs converge exponentially and animate the system to evolve towards the target persistently. The tracking error and closed-loop state signals are proved theoretically to be uniformly ultimately bounded (UUB) via Lyapunov–Krasovskii functional. A numerical simulation from the process industry verifies the effectiveness of the proposed method.

Time two-grid technique combined with temporal second order difference method for two-dimensional semilinear fractional sub-diffusion equations

In this paper, we construct a high order difference scheme for two-dimensional semilinear fractional sub-diffusion equations at first. To reduce the computation time, an efficient time two-grid algorithm is then proposed. The global convergence order of the two-grid scheme reaches O(τF2+τC4+h12+h22), where τF and τC represent the time-step sizes on the fine and coarse grids respectively, while h1 and h2 are the space-step sizes. Furthermore, stability and convergence of the scheme are carefully studied by some skills. At last, the theoretical statements are verified by numerical experiments.

Intermediate Sensory Feedback Assisted Multi-Step Neural Decoding for Reinforcement Learning Based Brain-Machine Interfaces.

Reinforcement-learning (RL)-based brain-machine interfaces (BMIs) interpret dynamic neural activity into movement intention without patients' real limb movements, which is promising for clinical applications. A movement task generally requires the subjects to reach the target within one step and rewards the subjects instantaneously. However, a real BMI scenario involves tasks that require multiple steps, during which sensory feedback is provided to indicate the status of the prosthesis, and the reward is only given at the end of the trial. Actually, subjects internally evaluate the sensory feedback to adjust motor activity. Existing RL-BMI tasks have not fully utilized the internal evaluation from the brain upon the sensory feedback to guide the decoder training, and there lacks an effective tool to assign credit for the multi-step decoding task. We propose first to extract intermediate guidance from the medial prefrontal cortex (mPFC) to assist the learning of multi-step decoding in an RL framework. To effectively explore the neural-action mapping in a large state-action space, a temporal difference (TD) method is incorporated into quantized attention-gated kernel reinforcement learning (QAGKRL) to assign the credit over the temporal sequence of movement, but also discriminate spatially in the Reproducing Kernel Hilbert Space (RKHS). We test our approach on the data collected from the primary motor cortex (M1) and the mPFC of rats when they brain control the cursor to reach the target within multiple steps. Compared with the models which only utilize the final reward, the intermediate evaluation interpreted from the mPFC can help improve the prediction accuracy by 10.9% on average across subjects, with faster convergence and more stability. Moreover, our proposed algorithm further increases 18.2% decoding accuracy compared with existing TD-RL methods. The results reveal the possibility of achieving better multi-step decoding performance for more complicated BMI tasks.

Kernel Temporal Difference based Reinforcement Learning for Brain Machine Interfaces.

Brain-machine interfaces (BMIs) enable people with disabilities to control external devices with their motor intentions through a decoder. Compared with supervised learning, reinforcement learning (RL) is more promising for the disabled because it can assist them to learn without actual limb movement. Current RL decoders deal with tasks with immediate reward delivery. But for tasks where the reward is only given by the end of the trial, existing RL methods may take a long time to train and are prone to becoming trapped in the local minima. In this paper, we propose to embed temporal difference method (TD) into Quantized Attention-Gated Kernel Reinforcement Learning (QAGKRL) to solve this temporal credit assignment problem. This algorithm utilizes a kernel network to ensure the global linear structure and adopts a softmax policy to efficiently explore the state-action mapping through TD error. We simulate a center-out task where the agent needs several steps to first reach a periphery target and then return to the center to get the external reward. Our proposed algorithm is tested on simulated data and compared with two state-of-the-art models. We find that introducing the TD method to QAGKRL achieves a prediction accuracy of 96.2% ± 0.77% (mean ± std), which is significantly better the other two methods.Clinical Relevance-This paper proposes a novel kernel temporal difference RL method for the multi-step task with delayed reward delivery, which potentially enables BMI online continuous decoding.

AKF-SR: Adaptive Kalman filtering-based successor representation

To understand animals’ behavior in finding relations between similar tasks and adapting themselves to changes in the tasks, it is necessary to know how the brain generalizes the learned knowledge from a previous task to unseen tasks. Recent studies in neuroscience suggest that Successor Representation (SR)-based models provide adaptation to changes in the goal locations or reward function faster than model-free algorithms, together with lower computational cost compared to that of model-based algorithms. However, it is not known how such representation might help animals to manage uncertainty in their decision making. Existing methods for the SR learning based on standard temporal difference methods (e.g., deep neural network-based algorithms) do not capture uncertainty about the estimated SR. In order to address this issue, the paper presents a Kalman filter-based SR framework, referred to as Adaptive Kalman Filtering-based Successor Representation (AKF–SR). First, Kalman temporal difference approach, which is a combination of Kalman filter and the temporal difference method, is used within the AKF–SR framework to cast the SR learning procedure into a filtering problem to benefit from uncertainty estimation of the SR, and also decreases in memory requirement and sensitivity to model’s parameters in comparison to deep neural network-based algorithms. An adaptive Kalman filtering approach is then applied within the proposed AKF–SR framework in order to tune the measurement noise covariance and measurement mapping function of Kalman filter as the most important parameters affecting the filter’s performance. Moreover, an active learning method that exploits the estimated uncertainty of the SR to form the behaviour policy leading to more visits to less certain values is proposed to improve the overall performance of an agent in terms of received rewards while interacting with its environment. Experimental results based on three reinforcement learning environments illustrate the efficacy of the proposed AKF–SR framework over state-of-the-art frameworks in terms of cumulative reward, reliability, time and computational cost, and speed of convergence to changes in the reward function.