Online Reinforcement Learning Research Articles

Eliminating Primacy Bias in Online Reinforcement Learning by Self-Distillation.

Excessive invalid explorations at the beginning of training lead deep reinforcement learning process to fall into the risk of overfitting, further resulting in spurious decisions, which obstruct agents in the following states and explorations. This phenomenon is termed primacy bias in online reinforcement learning. This work systematically investigates the primacy bias in online reinforcement learning, discussing the reason for primacy bias, while the characteristic of primacy bias is also analyzed. Besides, to learn a policy generalized to the following states and explorations, we develop an online reinforcement learning framework, termed self-distillation reinforcement learning (SDRL), based on knowledge distillation, allowing the agent to transfer the learned knowledge into a randomly initialized policy at regular intervals, and the new policy network is used to replace the original one in the following training. The core idea for this work is distilling knowledge from the trained policy to another policy can filter biases out, generating a more generalized policy in the learning process. Moreover, to avoid the overfitting of the new policy due to excessive distillations, we add an additional loss in the knowledge distillation process, using L2 regularization to improve the generalization, and the self-imitation mechanism is introduced to accelerate the learning on the current experiences. The results of several experiments in DMC and Atari 100k suggest the proposal has the ability to eliminate primacy bias for reinforcement learning methods, and the policy after knowledge distillation can urge agents to get higher scores more quickly.

Data-driven adaptive consensus control for heterogeneous nonlinear Multi-Agent Systems using online reinforcement learning

In this paper, a distributed control algorithm based on a data-driven approach is developed to solve the consensus control of heterogeneous nonlinear Multi-Agent Systems (MAS). The consensus obtained from the solution of the Hamilton–Jacobi–Bellman (HJB) equation is challenging for unknown nonlinear systems. To address this issue, improved online reinforcement learning (RL) is employed to generate an approximate solution for each agent to achieve consensus. Unlike model-based RL and traditional algorithms, this method leverages I/O data to guide the learning of policies without any prior knowledge of agent dynamics. Furthermore, the adaptability of algorithm to heterogeneous nonlinear agents is enhanced by implementing online updates to the control strategy and dynamic linearization (DL). The convergence analysis of the algorithm is provided, along with the impact of the learning rate parameters on the consensus of MAS. By comparing with other data-driven methods, simulations are conducted to verify the stability and adaptability of the proposed algorithm.

Adaptive optimal tracking control of networked linear systems under two-channel stochastic dropouts

This work investigates the adaptive optimal tracking control problem for networked discrete-time linear systems by directly using the data transmitted via communication networks. It is shown that the concerned problem can be addressed by solving two sub-problems: an adaptive optimal control one and an adaptive output regulation one. Two novel online reinforcement learning based policy iteration and value iteration algorithms are developed, which constitute an integrated framework to learn the optimal feedback control gain and the solutions to regulator equations by directly using the data transmitted via communication networks. Furthermore, it is shown that the tracking error of the closed-loop control system is mean square asymptotically stable even in the case of network packet stochastic dropouts. Simulation results demonstrate the efficacy of the proposed approaches.

Enhancing UAV Aerial Docking: A Hybrid Approach Combining Offline and Online Reinforcement Learning

In our study, we explore the task of performing docking maneuvers between two unmanned aerial vehicles (UAVs) using a combination of offline and online reinforcement learning (RL) methods. This task requires a UAV to accomplish external docking while maintaining stable flight control, representing two distinct types of objectives at the task execution level. Direct online RL training could lead to catastrophic forgetting, resulting in training failure. To overcome these challenges, we design a rule-based expert controller and accumulate an extensive dataset. Based on this, we concurrently design a series of rewards and train a guiding policy through offline RL. Then, we conduct comparative verification on different RL methods, ultimately selecting online RL to fine-tune the model trained offline. This strategy effectively combines the efficiency of offline RL with the exploratory capabilities of online RL. Our approach improves the success rate of the UAV’s aerial docking task, increasing it from 40% under the expert policy to 95%.

Mixed experience sampling for off-policy reinforcement learning

In deep reinforcement learning, experience replay is usually used to improve data efficiency and alleviate experience forgetting. However, online reinforcement learning is often influenced by the index of experience, which usually makes the phenomenon of unbalanced sampling. In addition, most experience replay methods ignore the differences among experiences, and cannot make full use of all experiences. Especially many “near”-policy experiences relatively relevant to the current policy are wasted, despite of the fact that they are beneficial for improving sample efficiency. This paper theoretically analyzes the influence of various factors on experience sampling, and then proposes a sampling method for experience replay based on frequency and similarity (FSER) to alleviate unbalanced sampling and increase the value of the sampled experiences. FSER prefers experiences that are rarely sampled or highly relevant to the current policy. FSER plays a critical role to balance the experience forgetting and wasting problems. Finally, FSER is combined with TD3 to achieve the state-of-the-art results in multiple tasks.

Safe adaptive output‐feedback optimal control of a class of linear systems

AbstractThe objective of this research is to enable safety‐critical systems to simultaneously learn and execute optimal control policies in a safe manner to achieve complex autonomy. Learning optimal policies via trial and error, that is, traditional reinforcement learning, is difficult to implement in safety‐critical systems, particularly when task restarts are unavailable. Safe model‐based reinforcement learning techniques based on a barrier transformation have recently been developed to address this problem. However, these methods rely on full‐state feedback, limiting their usability in a real‐world environment. In this work, an output‐feedback safe model‐based reinforcement learning technique based on a novel barrier‐aware dynamic state estimator has been designed to address this issue. The developed approach facilitates simultaneous learning and execution of safe control policies for safety‐critical linear systems. Simulation results indicate that barrier transformation is an effective approach to achieve online reinforcement learning in safety‐critical systems using output feedback.

Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resampling

Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resampling

A self‐learning human‐machine cooperative control method based on driver intention recognition

AbstractHuman‐machine cooperative control has become an important area of intelligent driving, where driver intention recognition and dynamic control authority allocation are key factors for improving the performance of cooperative decision‐making and control. In this paper, an online learning method is proposed for human‐machine cooperative control, which introduces a priority control parameter in the reward function to achieve optimal allocation of control authority under different driver intentions and driving safety conditions. Firstly, a two‐layer LSTM‐based sequence prediction algorithm is proposed to recognise the driver's lane change (LC) intention for human‐machine cooperative steering control. Secondly, an online reinforcement learning method is developed for optimising the steering authority to reduce driver workload and improve driving safety. The driver‐in‐the‐loop simulation results show that our method can accurately predict the driver's LC intention in cooperative driving and effectively compensate for the driver's non‐optimal driving actions. The experimental results on a real intelligent vehicle further demonstrate the online optimisation capability of the proposed RL‐based control authority allocation algorithm and its effectiveness in improving driving safety.

Symmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning

In deep reinforcement learning, estimating the value function to evaluate the quality of states and actions is essential. The value function is often trained using the least squares method, which implicitly assumes a Gaussian error distribution. However, a recent study suggested that the error distribution for training the value function is often skewed because of the properties of the Bellman operator, and violates the implicit assumption of normal error distribution in the least squares method. To address this, we proposed a method called Symmetric Q-learning, in which the synthetic noise generated from a zero-mean distribution is added to the target values to generate a Gaussian error distribution. We evaluated the proposed method on continuous control benchmark tasks in MuJoCo. It improved the sample efficiency of a state-of-the-art reinforcement learning method by reducing the skewness of the error distribution.

Robust Policy Learning via Offline Skill Diffusion

Skill-based reinforcement learning (RL) approaches have shown considerable promise, especially in solving long-horizon tasks via hierarchical structures. These skills, learned task-agnostically from offline datasets, can accelerate the policy learning process for new tasks. Yet, the application of these skills in different domains remains restricted due to their inherent dependency on the datasets, which poses a challenge when attempting to learn a skill-based policy via RL for a target domain different from the datasets' domains. In this paper, we present a novel offline skill learning framework DuSkill which employs a guided Diffusion model to generate versatile skills extended from the limited skills in datasets, thereby enhancing the robustness of policy learning for tasks in different domains. Specifically, we devise a guided diffusion-based skill decoder in conjunction with the hierarchical encoding to disentangle the skill embedding space into two distinct representations, one for encapsulating domain-invariant behaviors and the other for delineating the factors that induce domain variations in the behaviors. Our DuSkill framework enhances the diversity of skills learned offline, thus enabling to accelerate the learning procedure of high-level policies for different domains. Through experiments, we show that DuSkill outperforms other skill-based imitation learning and RL algorithms for several long-horizon tasks, demonstrating its benefits in few-shot imitation and online RL.

Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge

The problem of sample complexity of online reinforcement learning is often studied in the literature without taking into account any partial knowledge about the system dynamics that could potentially accelerate the learning process. In this paper, we study the sample complexity of online Q-learning methods when some prior knowledge about the dynamics is available or can be learned efficiently. We focus on systems that evolve according to an additive disturbance model of the form S_{h+1} = ƒ(S_h, A_h) + W_h, where ƒ represents the underlying system dynamics, and W_h are unknown disturbances independent of states and actions. In the setting of finite episodic Markov decision processes with S states, A actions, and episode length H, we present an optimistic Q-learning algorithm that achieves Õ(Poly(H)√T) regret under perfect knowledge of ƒ, where T is the total number of interactions with the system. This is in contrast to the typical Õ(Poly(H)√SAT) regret for existing Q-learning methods. Further, if only a noisy estimate ƒ_hat of ƒ is available, our method can learn an approximately optimal policy in a number of samples that is independent of the cardinalities of state and action spaces. The sub-optimality gap depends on the approximation error ƒ_hat − ƒ, as well as the Lipschitz constant of the corresponding optimal value function. Our approach does not require modeling of the transition probabilities and enjoys the same memory complexity as model-free methods.

Online Reinforcement Learning-Based Pedagogical Planning for Narrative-Centered Learning Environments

Pedagogical planners can provide adaptive support to students in narrative-centered learning environments by dynamically scaffolding student learning and tailoring problem scenarios. Reinforcement learning (RL) is frequently used for pedagogical planning in narrative-centered learning environments. However, RL-based pedagogical planning raises significant challenges due to the scarcity of data for training RL policies. Most prior work has relied on limited-size datasets and offline RL techniques for policy learning. Unfortunately, offline RL techniques do not support on-demand exploration and evaluation, which can adversely impact the quality of induced policies. To address the limitation of data scarcity and offline RL, we propose INSIGHT, an online RL framework for training data-driven pedagogical policies that optimize student learning in narrative-centered learning environments. The INSIGHT framework consists of three components: a narrative-centered learning environment simulator, a simulated student agent, and an RL-based pedagogical planner agent, which uses a reward metric that is associated with effective student learning processes. The framework enables the generation of synthetic data for on-demand exploration and evaluation of RL-based pedagogical planning. We have implemented INSIGHT with OpenAI Gym for a narrative-centered learning environment testbed with rule-based simulated student agents and a deep Q-learning-based pedagogical planner. Our results show that online deep RL algorithms can induce near-optimal pedagogical policies in the INSIGHT framework, while offline deep RL algorithms only find suboptimal policies even with large amounts of data.

UAV Trajectory, User Association, and Power Control for Multi-UAV-Enabled Energy-Harvesting Communications: Offline Design and Online Reinforcement Learning

UAV Trajectory, User Association, and Power Control for Multi-UAV-Enabled Energy-Harvesting Communications: Offline Design and Online Reinforcement Learning

Leader-Follower Based Online Reinforcement Learning Algorithm in Problem with Hierarchy Decision Makers

This paper presents an online integral reinforcement learning (RL) solution for problems with hierarchy decision makers. Specifically, we reformulate this model as a leader-follower game in which control input and deterministic disturbance act as decision makers at different levels of hierarchy: the control input plays the role of the leader while the disturbance plays the role of the follower. The main contributions of this paper can be summarized as follows. First, we introduce online RL to deal with systems that have partially unknown information, meaning that accurate dynamic information is not required. Second, we solve the leader-follower coupled Hamilton-Jacobi (HJ) and Riccati equations approximately online using the derived algorithm. Third, we provide turning laws for cost functions and controllers that ensure closed-loop stability simultaneously.

Off-policy RL algorithms can be sample-efficient for continuous control via sample multiple reuse

Sample efficiency is one of the most critical issues for online reinforcement learning (RL). Existing methods achieve higher sample efficiency by adopting model-based methods, Q-ensemble, or better exploration mechanisms. We, instead, propose to train an off-policy RL agent via updating on a fixed sampled batch multiple times, thus reusing these samples and better exploiting them within a single optimization loop. We name our method sample multiple reuse (SMR). We theoretically show the properties of Q-learning with SMR, e.g., convergence. Furthermore, we incorporate SMR with off-the-shelf off-policy RL algorithms and conduct experiments on a variety of continuous control benchmarks. Empirical results show that SMR significantly boosts the sample efficiency of the base methods across most of the evaluated tasks without any hyperparameter tuning or additional tricks.

Reinforcement Learning Building Control: An Online Approach With Guided Exploration Using Surrogate Models

Abstract The incorporation of emerging technologies, including solar photovoltaics, electric vehicles, battery energy storage, smart devices, Internet-of-Things devices, and sensors in buildings, desirable control objectives are becoming increasingly complex, calling for advanced controls approaches. Reinforcement learning (RL) is a powerful method for this. RL can adapt and learn from environmental interaction, but it can take a long time to learn and can be unstable initially due to limited environmental knowledge. In our research, we propose an online RL approach for buildings that uses data-driven surrogate models to guide the RL agent during its early training. This helps the controller learn faster and more stably than the traditional direct plug-and-learn online learning approach. In this research, we propose an online approach in buildings with RL where, with the help of data-driven surrogate models, the RL agent is guided during its early exploratory training stage, aiding the controller to learn a near-optimal policy faster and exhibiting more stable training progress than a traditional direct plug-and-learn online learning RL approach. The agents are assisted in their learning and action with information gained from the surrogate models generating multiple artificial trajectories starting from the current state. The research presented an exploration of various surrogate model-assisted training methods and revealed that models focusing on artificial trajectories around rule-based controls yielded the most stable performance. In contrast, models employing random exploration with a one-step look-ahead approach demonstrated superior overall performance.

Telemetry-aided cooperative multi-agent online reinforcement learning for DAG task scheduling in computing power networks

As demand for computing power and low latency in intelligence applications grows, the efficient management and coordination of resources in computing power networks become crucial. This paper presents a telemetry-aided multi-agent cooperation framework for DAG task scheduling in computing power networks. Utilizing distributed agents with network telemetry, the framework accurately assesses local network state information, formulates scheduling policies, and assigns tasks to edge servers. An online learning algorithm for DAG task scheduling is also introduced to enhance the cooperation strategy in decision-making, enabling rapid task scheduling and resource allocation decisions. Simulation results demonstrate a minimum 13.5% reduction in total task execution time compared to sub-optimal methods, along with improved node and link load balancing.

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems.

With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.

Practical Online Reinforcement Learning for Microprocessors with Micro-Armed Bandit

Practical Online Reinforcement Learning for Microprocessors with Micro-Armed Bandit

PTMB: An online satellite task scheduling framework based on pre-trained Markov decision process for multi-task scenario

The ever-growing scale of satellites and the increasing demand for Earth observation have led to significant interest in the problem of online multi-task scheduling for Earth observation satellites. Presently, task scheduling mostly relies on meta-heuristic and reinforcement learning algorithms. However, meta-heuristic algorithms possess slow convergence speed and are not suitable for multi-task scheduling scenarios, while reinforcement learning algorithms are unable to ensure solution quality due to unstable environmental states. In this study, we propose an innovative online micro-batch scheduling framework based on pre-trained reinforcement learning model (PTMB). This framework splits the satellite scheduling issue into two phases: task decision-making and task allocation. We leverage the micro-batch processing mode and introduce a pre-trained Markov decision model during the task decision-making phase. Additionally, we incorporate resource pre-allocation, task sequencing, task order shuffle, and other strategies to enhance the overall solution quality. Simulation experiments reveal a significant enhancement in performance of our proposed method. Specifically, when dealing with task sizes of 100 and 300, the task scheduling reward of the proposed framework surpasses that of the improved genetic algorithm by 11.5% and 0.4%, respectively, while reducing time consumption by 97.8% and 99.3%. Furthermore, our framework surpasses online reinforcement learning scheduling method, which is also based on Markov decision process model, achieving improvements of 55.1%, 5.6%, and 15.6% in task scheduling reward and reductions in time consumed by 94.9%, 96.7%, and 99.1% at task sizes of 100, 300, and 500, respectively.