Model-based Bayesian Reinforcement Learning Research Articles

Model-based Bayesian reinforcement learning for enhancing primary user performance under jamming attack

Due to the broadcast nature of wireless networks, jamming attacks are inevitable. Therefore, this paper considers enhancing the throughput of a primary user (PU) suffering from a random jamming attack near the receiver in a cognitive radio network. A cooperative spectrum-sharing scheme is proposed between the PU and an energy-constrained secondary user (SU) harvesting non-RF energy. The SU location is optimized to maximize the PU throughput. Then, a decision problem is formulated to adaptively optimize the PU decision regarding the cooperation with SU and maximize the PU throughput in the long run, considering the uncertainties about the jammer activity and the SU’s available energy. Unlike the previous work, our scheme assumes no prior knowledge about the parameters controlling the jammer’s random behavior, the energy harvesting process, and the sensing system. Bayesian reinforcement learning (BRL) is adopted to learn the unknown parameters and optimize the PU decisions simultaneously. The Bayes-adaptive mixed-observable Markov decision process (BAMOMDP) model is proposed to extend the current BRL models to handle the state’s mixed observability. Simulation results show the efficiency and robustness of the proposed scheme in enhancing the PU/SU throughputs and its ability to adapt to non-stationary environments. Compared with previous work, the proposed scheme can improve the PU average throughput by 15% in non-stationary environments and up to 40% in stationary environments with low jamming sensing capabilities.

SACBP: Belief space planning for continuous-time dynamical systems via stochastic sequential action control

We propose a novel belief space planning technique for continuous dynamics by viewing the belief system as a hybrid dynamical system with time-driven switching. Our approach is based on the perturbation theory of differential equations and extends sequential action control to stochastic dynamics. The resulting algorithm, which we name SACBP, does not require discretization of spaces or time and synthesizes control signals in near real-time. SACBP is an anytime algorithm that can handle general parametric Bayesian filters under certain assumptions. We demonstrate the effectiveness of our approach in an active sensing scenario and a model-based Bayesian reinforcement learning problem. In these challenging problems, we show that the algorithm significantly outperforms other existing solution techniques including approximate dynamic programming and local trajectory optimization.

Approximate Bayesian reinforcement learning based on estimation of plant

This study proposes an approximate parametric model-based Bayesian reinforcement learning approach for robots, based on online Bayesian estimation and online planning for an estimated model. The proposed approach is designed to learn a robotic task with a few real-world samples and to be robust against model uncertainty, within feasible computational resources. The proposed approach employs two-stage modeling, which is composed of (1) a parametric differential equation model with a few parameters based on prior knowledge such as equations of motion, and (2) a parametric model that interpolates a finite number of transition probability models for online estimation and planning. The proposed approach modifies the online Bayesian estimation to be robust against approximation errors of the parametric model to a real plant. The policy planned for the interpolating model is proven to have a form of theoretical robustness. Numerical simulation and hardware experiments of a planar peg-in-hole task demonstrate the effectiveness of the proposed approach.

An Approximate Bayesian Reinforcement Learning Approach Using Robust Control Policy and Tree Search

For autonomous robots, we propose an approximate model-based Bayesian reinforcement learning (MB-BRL) approach that reduces real-world samples within feasible computational efforts. Firstly, to find an approximate solution of an original undiscounted infinite horizon MB-BRL problem with a cost-free termination, we consider a finite horizon (FH) MB-BRL problem in which terminal costs are given by robust control policies. The resulting performance is better than or equal to the performance obtained with a robust method, while the resulting policy may choose an explorative behavior to get useful information about parametric model uncertainty for reducing real-world samples. Secondly, to obtain a feasible solution of the FH MB-BRL problem using simulation samples, we propose a combination of robust RL, Monte Carlo tree search (MCTS), and Bayesian inference. We show an idea of reusing previous MCTS samples for Bayesian inference at a leaf node. The proposed approach allows an agent to choose from multiple robust policies at a leaf node. Numerical experiments of a two-dimensional peg-in-hole task demonstrate the effectiveness of the proposed approach.

Tighter Value Function Bounds for Bayesian Reinforcement Learning

Bayesian reinforcement learning (BRL) provides a principled framework for optimal exploration-exploitation tradeoff in reinforcement learning. We focus on model based BRL, which involves a compact formulation of the optimal tradeoff from the Bayesian perspective. However, it still remains a computational challenge to compute the Bayes-optimal policy. In this paper, we propose a novel approach to compute tighter value function bounds of the Bayes-optimal value function, which is crucial for improving the performance of many model-based BRL algorithms. We then present how our bounds can be integrated into real-time AO* heuristic search, and provide a theoretical analysis on the impact of improved bounds on the search efficiency. We also provide empirical results on standard BRL domains that demonstrate the effectiveness of our approach.

Reward Shaping for Model-Based Bayesian Reinforcement Learning

Bayesian reinforcement learning (BRL) provides a formal framework for optimal exploration-exploitation tradeoff in reinforcement learning. Unfortunately, it is generally intractable to find the Bayes-optimal behavior except for restricted cases. As a consequence, many BRL algorithms, model-based approaches in particular, rely on approximated models or real-time search methods. In this paper, we present potential-based shaping for improving the learning performance in model-based BRL. We propose a number of potential functions that are particularly well suited for BRL, and are domain-independent in the sense that they do not require any prior knowledge about the actual environment. By incorporating the potential function into real-time heuristic search, we show that we can significantly improve the learning performance in standard benchmark domains.

Model-based Bayesian Reinforcement Learning in Factored Markov Decision Process

Learning the enormous number of parameters is a challenging problem in model-based Bayesian reinforcement learning. In order to solve the problem, we propose a model-based factored Bayesian reinforcement learning (F-BRL) approach. F-BRL exploits a factored representation to describe states to reduce the number of parameters. Representing the conditional independence relationships between state features using dynamic Bayesian networks, F-BRL adopts Bayesian inference method to learn the unknown structure and parameters of the Bayesian networks simultaneously. A point-based online value iteration approach is then used for planning and learning online. The experimental and simulation results show that the proposed approach can effectively reduce the number of learning parameters, and enable online learning for dynamic systems with thousands of states.