Modern Reinforcement Learning Research Articles

Modeling Bellman-error with logistic distribution with applications in reinforcement learning

In modern Reinforcement Learning (RL) approaches, optimizing the Bellman error is a critical element across various algorithms, notably in deep Q-Learning and related methodologies. Traditional approaches predominantly employ the mean-squared Bellman error (MSELoss) as the standard loss function. However, the assumption of Bellman errors following the Gaussian distribution may oversimplify the nuanced characteristics of RL applications. In this work, we revisit the distribution of Bellman error in RL training, demonstrating that it tends to follow the Logistic distribution rather than the commonly assumed Normal distribution. We propose replacing MSELoss with a Logistic maximum likelihood function (LLoss) and rigorously test this hypothesis through extensive numerical experiments across diverse online and offline RL environments. Our findings consistently show that integrating the Logistic correction into the loss functions of various baseline RL methods leads to superior performance compared to their MSE counterparts. Additionally, we employ Kolmogorov–Smirnov tests to substantiate that the Logistic distribution offers a more accurate fit for approximating Bellman errors. This study also offers a novel theoretical contribution by establishing a clear connection between the distribution of Bellman error and the practice of proportional reward scaling, a common technique for performance enhancement in RL. Moreover, we explore the sample-accuracy trade-off involved in approximating the Logistic distribution, leveraging the Bias–Variance decomposition to mitigate excessive computational resources. The theoretical and empirical insights presented in this study lay a significant foundation for future research, potentially advancing methodologies, and understanding in RL, particularly in the distribution-based optimization of Bellman error.

SWOAM: Swarm optimized agents for energy management in grid-interactive connected buildings

With increasing complexity of smart city energy systems and rising energy demand, effective energy management solutions are crucial. Buildings now incorporate renewable energy sources and battery storage for efficient energy utilization, making optimal control strategies important. Compared to rule-based controllers and model-based methods, swarm and evolutionary algorithms have the advantages of providing cost-effective, stable, and scalable alternatives. However, their potential in data-rich environments and multi-building energy systems remains underexplored. This research bridges this gap by demonstrating the cooperative capabilities of population-based optimization agents for efficient energy management. Specifically, novel algorithm and framework, named Swarm Optimized Agents for Sequential Decision Making (SWOAM) is proposed. It combines an online k-means classifier and a swarm optimizer for optimal control of multi-building energy systems. By designing an online K-means learning strategy and an agent-based sequential episodic sampling approach, it is feasible to train a swarm of agents find an optimal energy management policy for buildings in a district. For training and evaluation, we use the standardized CityLearn building energy management environment. The agents are evaluated on three key metrics: electricity cost, carbon dioxide emissions and grid ramping. SWOAM delivers state-of-the-art performance and outperforms modern reinforcement learning and rule-based controllers.

Graph Representations for Reinforcement Learning

Graph analysis is becoming increasingly important due to the expressive power of graph models and the efficient algorithms available for processing them. Reinforcement Learning is one domain that could benefit from advancements in graph analysis, given that a learning agent may be integrated into an environment that can be represented as a graph. Nevertheless, the structural irregularity of graphs and the lack of prior labels make it difficult to integrate such a model into modern Reinforcement Learning frameworks that rely on artificial neural networks. Graph embedding enables the learning of low-dimensional vector representations that are more suited for machine learning algorithms, while retaining essential graph features. This paper presents a framework for evaluating graph embedding algorithms and their ability to preserve the structure and relevant features of graphs by means of an internal validation metric, without resorting to subsequent tasks that require labels for training. Based on this framework, three defined algorithms that meet the necessary requirements for solving a specific problem of Reinforcement Learning in graphs are selected, analyzed, and compared. These algorithms are Graph2Vec, GL2Vec, and Wavelet Characteristics, with the latter two demonstrating superior performance.

Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model

This paper studies a central issue in modern reinforcement learning, the sample efficiency, and makes progress toward solving an idealistic scenario that assumes access to a generative model or a simulator. Despite a large number of prior works tackling this problem, a complete picture of the trade-offs between sample complexity and statistical accuracy has yet to be determined. In particular, all prior results suffer from a severe sample size barrier in the sense that their claimed statistical guarantees hold only when the sample size exceeds some enormous threshold. The current paper overcomes this barrier and fully settles this problem; more specifically, we establish the minimax optimality of the model-based approach for any given target accuracy level. To the best of our knowledge, this work delivers the first minimax-optimal guarantees that accommodate the entire range of sample sizes (beyond which finding a meaningful policy is information theoretically infeasible).

Provably Efficient Reinforcement Learning with Linear Function Approximation

Modern reinforcement learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy. The introduction of function approximation raises a fundamental set of challenges involving computational and statistical efficiency, especially given the need to manage the exploration/exploitation trade-off. As a result, a core RL question remains open: how can we design provably efficient RL algorithms that incorporate function approximation? This question persists even in a basic setting with linear dynamics and linear rewards, for which only linear function approximation is needed. This paper presents the first provable RL algorithm with both polynomial run time and polynomial sample complexity in this linear setting, without requiring a “simulator” or additional assumptions. Concretely, we prove that an optimistic modification of least-squares value iteration—a classical algorithm frequently studied in the linear setting—achieves [Formula: see text] regret, where d is the ambient dimension of feature space, H is the length of each episode, and T is the total number of steps. Importantly, such regret is independent of the number of states and actions. Funding: This work was supported by the Defense Advanced Research Projects Agency program on Lifelong Learning Machines.

Softmax policy gradient methods can take exponential time to converge

AbstractThe softmax policy gradient (PG) method, which performs gradient ascent under softmax policy parameterization, is arguably one of the de facto implementations of policy optimization in modern reinforcement learning. For$$\gamma $$γ-discounted infinite-horizon tabular Markov decision processes (MDPs), remarkable progress has recently been achieved towards establishing global convergence of softmax PG methods in finding a near-optimal policy. However, prior results fall short of delineating clear dependencies of convergence rates on salient parameters such as the cardinality of the state space$${\mathcal {S}}$$Sand the effective horizon$$\frac{1}{1-\gamma }$$11-γ, both of which could be excessively large. In this paper, we deliver a pessimistic message regarding the iteration complexity of softmax PG methods, despite assuming access to exact gradient computation. Specifically, we demonstrate that the softmax PG method with stepsize$$\eta $$ηcan take$$\begin{aligned} \frac{1}{\eta } |{\mathcal {S}}|^{2^{\Omega \big (\frac{1}{1-\gamma }\big )}} ~\text {iterations} \end{aligned}$$1η|S|2Ω(11-γ)iterationsto converge, even in the presence of a benign policy initialization and an initial state distribution amenable to exploration (so that the distribution mismatch coefficient is not exceedingly large). This is accomplished by characterizing the algorithmic dynamics over a carefully-constructed MDP containing only three actions. Our exponential lower bound hints at the necessity of carefully adjusting update rules or enforcing proper regularization in accelerating PG methods.

Exploring the Integration of Lisp into a Modern Reinforcement Learning Project Through the Use of Hy

This paper explores the usage of Lisp in a small modern Reinforcement Learning (RL) project. The Lisp dialect, Hy programming language, is used to incorporate the traditional libraries and packages in up-to-date workflows. This project is centered around the usage of NetHack for RL. The MiniHack sandbox framework and NetHack Learning Environment (NLE) are used to create custom training/testing environments and tasks. The MiniHack sandbox framework creates a simple level editor and creation interface for use in the training and evaluation process of the agent. NLE is chosen as the working environment. For the agent model, this project adopts Torchbeast’s PolyBeast, a PyTorch implementation of the IMPALA architecture. The usage of Hy within this project is forefront, and so it is implemented as much as possible to accomplish the tasks.

Co-Evolution of Predator-Prey Ecosystems by Reinforcement Learning Agents.

The problem of finding adequate population models in ecology is important for understanding essential aspects of their dynamic nature. Since analyzing and accurately predicting the intelligent adaptation of multiple species is difficult due to their complex interactions, the study of population dynamics still remains a challenging task in computational biology. In this paper, we use a modern deep reinforcement learning (RL) approach to explore a new avenue for understanding predator-prey ecosystems. Recently, reinforcement learning methods have achieved impressive results in areas, such as games and robotics. RL agents generally focus on building strategies for taking actions in an environment in order to maximize their expected returns. Here we frame the co-evolution of predators and preys in an ecosystem as allowing agents to learn and evolve toward better ones in a manner appropriate for multi-agent reinforcement learning. Recent significant advancements in reinforcement learning allow for new perspectives on these types of ecological issues. Our simulation results show that throughout the scenarios with RL agents, predators can achieve a reasonable level of sustainability, along with their preys.

HEVC Rate-Distortion Optimization with Source Modeling

The Rate-Distortion adaptive mechanisms of MPEG-HEVC (High Efficiency Video Coding) and its derivatives are an incremental improvement in the software reference encoder, providing a selective Lagrangian parameter choice which varies by encoding mode (intra or inter) and picture reference level. Since this weighting factor (and the balanced cost functions it impacts) are crucial to the RD optimization process, affecting several encoder decisions and both coding efficiency and quality of the encoded stream, we investigate an improvement by modern reinforcement learning methods. We develop a neural-based agent that learns a real-valued control policy to maximize rate savings by input signal pattern, mapping pixel intensity values from the picture at the coding tree unit level, to the appropriate weighting-parameter. Our testing on reference software yields improvements for coding efficiency performance across different video sequences, in multiple classes of video.

Designing Human-Autonomy Teaming Experiments Through Reinforcement Learning

This paper creates and defines a framework for building and implementing human-autonomy teaming experiments that enable the utilization of modern reinforcement learning models. These models are used to train artificial agents to then interact alongside humans in a human-autonomy team. The framework was synthesized from experience gained redesigning a previously known and validated team task simulation environment known as NeoCITIES. Through this redesign, several important high-level distinctions were made that regarded both the artificial agent and the task simulation itself. The distinctions within the framework include gamification, access to high-performance computing, a proper reward function, an appropriate team task simulation, and customizability. This framework enables researchers to create experiments that are more usable for the human and more closely resemble real-world human-autonomy interactions. The framework also allows researchers to create veritable and robust experimental platforms meant to study human-autonomy teaming for years to come.

Control of neural systems at multiple scales using model-free, deep reinforcement learning

Recent improvements in hardware and data collection have lowered the barrier to practical neural control. Most of the current contributions to the field have focus on model-based control, however, models of neural systems are quite complex and difficult to design. To circumvent these issues, we adapt a model-free method from the reinforcement learning literature, Deep Deterministic Policy Gradients (DDPG). Model-free reinforcement learning presents an attractive framework because of the flexibility it offers, allowing the user to avoid modeling system dynamics. We make use of this feature by applying DDPG to models of low-level and high-level neural dynamics. We show that while model-free, DDPG is able to solve more difficult problems than can be solved by current methods. These problems include the induction of global synchrony by entrainment of weakly coupled oscillators and the control of trajectories through a latent phase space of an underactuated network of neurons. While this work has been performed on simulated systems, it suggests that advances in modern reinforcement learning may enable the solution of fundamental problems in neural control and movement towards more complex objectives in real systems.

True online temporal-difference learning

The temporal-difference methods TD(λ) and Sarsa(λ) form a core part of modern reinforcement learning. Their appeal comes from their good performance, low computational cost, and their simple interpretation, given by their forward view. Recently, new versions of these methods were introduced, called true online TD(λ) and true online Sarsa(λ), respectively (van Seijen & Sutton, 2014). Algorithmically, these true online methods only make two small changes to the update rules of the regular methods, and the extra computational cost is negligible in most cases. However, they follow the ideas underlying the forward view much more closely. In particular, they maintain an exact equivalence with the forward view at all times, whereas the traditional versions only approximate it for small step-sizes. We hypothesize that these true online methods not only have better theoretical properties, but also dominate the regular methods empirically. In this article, we put this hypothesis to the test by performing an extensive empirical comparison. Specifically, we compare the performance of true online TD(λ)/Sarsa(λ) with regular TD(λ)/Sarsa(λ) on random MRPs, a real-world myoelectric prosthetic arm, and a domain from the Arcade Learning Environment. We use linear function approximation with tabular, binary, and non-binary features. Our results suggest that the true online methods indeed dominate the regular methods. Across all domains/representations the learning speed of the true online methods are often better, but never worse than that of the regular methods. An additional advantage is that no choice between traces has to be made for the true online methods. Besides the empirical results, we provide an in-dept analysis of the theory behind true online temporal-difference learning. In addition, we show that new true online temporal-difference methods can be derived by making changes to the online forward view and then rewriting the update equations.

Planning for uncertainty

Planning for uncertainty

Dopamine, Reinforcement Learning, and Addiction

Dopamine is intimately linked with the modes of action of drugs of addiction. However, although its role in the initiation of drug abuse seems relatively uncomplicated, its possible involvement in the development of compulsive drug taking, and indeed vulnerability and relapse, is less clear. We first describe a modern reinforcement learning view of affective control, focusing on the roles for dopamine. We then use this as a framework to sketch various notions of the neuromodulator's possible participation in initiation and compulsion. We end with some pointers towards future theoretical developments.

Can the apparent adaptation of dopamine neurons’ mismatch sensitivities be reconciled with their computation of reward prediction errors?

According to modern reinforcement learning theories, midbrain dopamine (DA) neurons are part of an adaptive system within which learned expectations filter reward-related signals to enable computation of reward prediction errors (RPEs). Recent electrophysiological data on DA neuron responses to probabilistic reward schedules inspired the idea that DA neurons might be adapting their mismatch sensitivities to reflect variances of expected rewards. Taken literally as a mathematical hypothesis, this idea contradicts reinforcement learning theory, and most computational models of basal ganglia learning. Here, we report a qualitative mathematical derivation of the implications of a generic class of circuit models for learning to compute RPEs. This analysis and concordant circuit simulations, both of which predict DA neuron responses on probabilistic schedules, support a reinterpretation of the electrophysiological data that is fully compatible with the examined class of RPE models. This reinterpretation implies a novel and readily testable prediction.