Off-policy Reinforcement Learning Algorithm Research Articles

Quadratic Tracking Control of Linear Stochastic Systems with Unknown Dynamics Using Average Off-Policy Q-Learning Method

This article investigates the optimal tracking control problem for data-based stochastic discrete-time linear systems. An average off-policy Q-learning algorithm is proposed to solve the optimal control problem with random disturbances. Compared with the existing off-policy reinforcement learning (RL) algorithm, the proposed average off-policy Q-learning algorithm avoids the assumption of an initial stability control. First, a pole placement strategy is used to design an initial stable control for systems with unknown dynamics. Second, the initial stable control is used to design a data-based average off-policy Q-learning algorithm. Then, this algorithm is used to solve the stochastic linear quadratic tracking (LQT) problem, and a convergence proof of the algorithm is provided. Finally, numerical examples show that this algorithm outperforms other algorithms in a simulation.

Heterogeneous unmanned swarm formation containment control based on reinforcement learning

In response to the problem of time-varying spherical formation control for a heterogeneous unmanned aerial vehicle (UAV) swarm system with dynamic uncertainty in the system model, this paper proposes an optimal distributed formation containment control method based on reinforcement learning. By combining the time-varying formation containment vector design with the distributed predefined time observer and constructing the augmented system of a multi-quadrotor UAV system and an observer, the problem of time-varying formation containment control in heterogeneous swarm systems is transformed into a stabilization problem. By introducing a value function with a discount factor, the stabilization problem of a heterogeneous UAV swarm system is transformed into an optimal control problem. Using the “actor-critic” neural network, combined with an off-policy reinforcement learning algorithm and distributed control methods, the solution to the formation containment controller is achieved in a data-driven manner. The stability of distributed observer and formation containment controller as well as the convergence of reinforcement learning algorithms are demonstrated using Lyapunov and related theory. The numerical simulation results demonstrate that the observation error of the designed distributed observer converges within a predefined time of 0.1 s, while the overall formation tracking error of the heterogeneous swarm converges within 2.74 s, thereby further validating the effectiveness and superiority of the designed control scheme.

Integrating an Ensemble Reward System into an Off-Policy Reinforcement Learning Algorithm for the Economic Dispatch of Small Modular Reactor-Based Energy Systems

Nuclear Integrated Energy Systems (NIES) have emerged as a comprehensive solution for navigating the changing energy landscape. They combine nuclear power plants with renewable energy sources, storage systems, and smart grid technologies to optimize energy production, distribution, and consumption across sectors, improving efficiency, reliability, and sustainability while addressing challenges associated with variability. The integration of Small Modular Reactors (SMRs) in NIES offers significant benefits over traditional nuclear facilities, although transferring involves overcoming legal and operational barriers, particularly in economic dispatch. This study proposes a novel off-policy Reinforcement Learning (RL) approach with an ensemble reward system to optimize economic dispatch for nuclear-powered generation companies equipped with an SMR, demonstrating superior accuracy and efficiency when compared to conventional methods and emphasizing RL’s potential to improve NIES profitability and sustainability. Finally, the research attempts to demonstrate the viability of implementing the proposed integrated RL approach in spot energy markets to maximize profits for nuclear-driven generation companies, establishing NIES’ profitability over competitors that rely on fossil fuel-based generation units to meet baseload requirements.

Integrating human learning and reinforcement learning: A novel approach to agent training

Off-policy reinforcement learning (RL) algorithms are known for improving sample efficiency by employing prior experiences in experience replay memory. However, most existing off-policy RL algorithms are bottlenecked by the slow convergence speed and demand of a large number of interaction samples. In contrast, on-policy RL algorithms converge fast and continuously generate new samples. However, their success heavily depends on the accuracy of the generated samples. To address these challenges, a novel RL framework called HI-FER (Human-learning Inspired Frequent Experience Replay) is proposed by mimicking the process of human learning. HI-FER employs a parallelized experience replay and repetitive training framework to expedite the convergence rate of off-policy algorithms, which imitates the function of the human brain’s parallel information processing and repetitive learning. Additionally, a periodic network reset strategy and dynamic memory updating are leveraged by imitating the forgetting mechanism of humans to prevent overfitting triggered by repetitive updating on limited experiences. Extensive comparison experiments and ablation studies are performed on benchmark environments to evaluate the proposed method. The empirical results demonstrate that HI-FER outperforms the baselines in terms of sample efficiency on state-based (14% improvements) and image-based (51% improvements) tasks from DMControl. Project website and code: https://github.com/Arya87/HI-FER.

Utilizing reinforcement learning for de novo drug design

AbstractDeep learning-based approaches for generating novel drug molecules with specific properties have gained a lot of interest in the last few years. Recent studies have demonstrated promising performance for string-based generation of novel molecules utilizing reinforcement learning. In this paper, we develop a unified framework for using reinforcement learning for de novo drug design, wherein we systematically study various on- and off-policy reinforcement learning algorithms and replay buffers to learn an RNN-based policy to generate novel molecules predicted to be active against the dopamine receptor DRD2. Our findings suggest that it is advantageous to use at least both top-scoring and low-scoring molecules for updating the policy when structural diversity is essential. Using all generated molecules at an iteration seems to enhance performance stability for on-policy algorithms. In addition, when replaying high, intermediate, and low-scoring molecules, off-policy algorithms display the potential of improving the structural diversity and number of active molecules generated, but possibly at the cost of a longer exploration phase. Our work provides an open-source framework enabling researchers to investigate various reinforcement learning methods for de novo drug design.

Optimal Tracking Control of Heterogeneous MASs Using Event-Driven Adaptive Observer and Reinforcement Learning.

This article considers the output tracking control problem of nonidentical linear multiagent systems (MASs) using a model-free reinforcement learning (RL) algorithm, where partial followers have no prior knowledge of the leader's information. To lower the communication and computing burden among agents, an event-driven adaptive distributed observer is proposed to predict the leader's system matrix and state, which consists of the estimated value of relative states governed by an edge-based predictor. Meanwhile, the integral input-based triggering condition is exploited to decide whether to transmit its private control input to its neighbors. Then, an RL-based state feedback controller for each agent is developed to solve the output tracking control problem, which is further converted into the optimal control problem by introducing a discounted performance function. Inhomogeneous algebraic Riccati equations (AREs) are derived to obtain the optimal solution of AREs. An off-policy RL algorithm is used to learn the solution of inhomogeneous AREs online without requiring any knowledge of the system dynamics. Rigorous analysis shows that under the proposed event-driven adaptive observer mechanism and RL algorithm, all followers are able to synchronize the leader's output asymptotically. Finally, a numerical simulation is demonstrated to verify the proposed approach in theory.

Double Buffers CEM-TD3: More Efficient Evolution and Richer Exploration

CEM-TD3 is a combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (TD3), and it achieves a satisfactory trade-off between performance and sample efficiency. However, we find that CEM-TD3 cannot fully address the low efficiency of policy search caused by CEM, and the policy gradient learning introduced by TD3 will weaken the diversity of individuals in the population. In this paper, we propose Double Buffers CEM-TD3 (DBCEM-TD3) that optimizes both CEM and TD3. For CEM, DBCEM-TD3 maintains an actor buffer to store the population required for evolution. In each iteration, it only needs to generate a small number of actors to replace the poor actors in the policy buffer to achieve more efficient evolution. The fitness of individuals in the actor buffer decreases exponentially with time, which can avoid premature convergence of the mean actor. For TD3, DBCEM-TD3 maintains a critic buffer with the same number of critics as the number of actors generated in each iteration, and each critic is trained independently by sampling from the shared replay buffer. In each iteration, each newly generated actor uses different critics to guide learning. This ensures more diverse behaviors among the learned actors, enabling richer experiences to be collected during the evaluation phase. We conduct experimental evaluations on five continuous control tasks provided by OpenAI Gym. DBCEM-TD3 outperforms CEM-TD3, TD3, and other classic off-policy reinforcement learning algorithms in terms of performance and sample efficiency.

Robust hierarchical games of linear discrete-time systems based on off-policy model-free reinforcement learning

An off-policy model-free reinforcement learning (RL) algorithm is proposed for a robust hierarchical game while considering incomplete information and input constraints. The robust hierarchical game exhibits characteristics of a Stackelberg–Nash (SN) game, where equilibrium points are designated as Stackelberg–Nash–Saddle equilibrium (SNE) points. An off-policy method is employed for the RL algorithm, addressing input constraints by using excitation input instead of real-time update polices as control inputs. Moreover, a model-free method is implemented for the off-policy RL algorithm, accounting for the challenge posed by incomplete information. The goal of this paper is to develop an off-policy model-free RL algorithm to obtain approximate SNE polices of the robust hierarchical game with incomplete information and input constraints. Furthermore, the convergence and effectiveness of the off-policy model-free RL algorithm are guaranteed by proving the equivalence of Bellman equation between nominal SNE policies and approximate SNE policies. Finally, a simulation is provided to verify the advantage of the developed algorithm.

Distributed Minmax Strategy for Multiplayer Games: Stability, Robustness, and Algorithms.

This article studies a distributed minmax strategy for multiplayer games and develops reinforcement learning (RL) algorithms to solve it. The proposed minmax strategy is distributed, in the sense that it finds each player's optimal control policy without knowing all the other players' policies. Each player obtains its distributed control policy by solving a distributed algebraic Riccati equation in a multiplayer noncooperative game. This policy is found against the worst policies of all the other players. We guarantee the existence of distributed minmax solutions and study their L2 and asymptotic stabilities. Under mild conditions, the resulting minmax control policies are shown to improve robust gain and phase margins of multiplayer systems compared to the standard linear-quadratic regulator controller. Distributed minmax solutions are found using both model-based policy iteration and data-driven off-policy RL algorithms. Simulation examples verify the proposed formulation and its computational efficiency over the nondistributed Nash solutions.

Off-policy RL algorithms can be sample-efficient for continuous control via sample multiple reuse

Sample efficiency is one of the most critical issues for online reinforcement learning (RL). Existing methods achieve higher sample efficiency by adopting model-based methods, Q-ensemble, or better exploration mechanisms. We, instead, propose to train an off-policy RL agent via updating on a fixed sampled batch multiple times, thus reusing these samples and better exploiting them within a single optimization loop. We name our method sample multiple reuse (SMR). We theoretically show the properties of Q-learning with SMR, e.g., convergence. Furthermore, we incorporate SMR with off-the-shelf off-policy RL algorithms and conduct experiments on a variety of continuous control benchmarks. Empirical results show that SMR significantly boosts the sample efficiency of the base methods across most of the evaluated tasks without any hyperparameter tuning or additional tricks.

Model-free control of underwater vehicle-manipulator system interacting with unknown environments

The underwater vehicle-manipulator (UVM) interaction tasks require that UVM can adapt to different environments and have good stability properties. However, the complicated structures of UVM and uncertainties of the environment lead to poor interaction. In addition, transient performance related to overshoot and convergence is required during the interaction to ensure the safety of the UVM and environment. In this paper, an inner-loop/outer-loop control formulation is developed for underwater vehicle-manipulator interacting with environment, subject to unknown system and environment dynamic. For the inner-loop, an approximation-free state feedback controller is designed for UVM system without a prior knowledge of dynamics, capable of guaranteeing trajectory tracking with prescribed performance. A prescribed function is constructed, which allows the convergence time and convergence accuracy to be predetermined. Moreover, it is independent of the initial conditions. For the outer-loop, an off-policy reinforcement learning algorithm is presented to optimize the parameters of the admittance model without requiring the information of environment dynamic. The proposed admittance optimization is feasible for arbitrary desired trajectory. Finally, simulation and experimental results validate the effectiveness of the proposed model-free tracking controller.

Model-Free Quantum Gate Design and Calibration Using Deep Reinforcement Learning

High-fidelity quantum gate design is important for various quantum technologies, such as quantum computation and quantum communication. Numerous control policies for quantum gate design have been proposed given a dynamical model of the quantum system of interest. However, a quantum system is often highly sensitive to noise, and obtaining its accurate modeling can be difficult for many practical applications. Thus, the control policy based on a quantum system model may be unpractical for quantum gate design. Also, quantum measurements collapse quantum states, which makes it challenging to obtain information through measurements during the control process. In this paper, we propose a novel training framework using deep reinforcement learning for model-free quantum control. The proposed framework relies only on the measurement at the end of the control process and offers the ability to find the optimal control policy without access to quantum systems during the learning process. The effectiveness of the proposed technique is numerically demonstrated for model-free quantum gate design and quantum gate calibration using off-policy reinforcement learning algorithms.

An Off-Policy Reinforcement Learning Algorithm Customized for Multi-Task Fusion in Large-Scale Recommender Systems

An Off-Policy Reinforcement Learning Algorithm Customized for Multi-Task Fusion in Large-Scale Recommender Systems

Scalable reinforcement learning approaches for dynamic pricing in ride-hailing systems

Dynamic pricing is a widely applied strategy by ride-hailing companies, such as Uber and Lyft, to match the trip demand with the availability of drivers. Deciding proper pricing policies is challenging and existing reinforcement learning (RL)-based solutions are restricted in solving small-scale problems. In this study, we contribute to RL-based approaches that can address the dynamic pricing problem in real-world-scale ride-hailing systems. We first characterize the dynamic pricing problem with a clear distinction between historical prices and current prices. We then translate our dynamic pricing problem into Markov Decision Process (MDP) and prove the existence of a deterministic stationary optimal policy. Our solutions are based on an off-policy reinforcement learning algorithm called twin-delayed deep determinant policy gradient (TD3) that performs offline learning of the optimal pricing policy using historical data and applies the learned policy to the next time slot, e.g., one week. We enhance TD3 by creating three mechanisms to reduce our model complexity and enhance training effectiveness. Extensive numerical experiments are conducted on both small grid networks (16 zones) and the NYC network (242 zones) to demonstrate the performance of the proposed algorithm. The results show our algorithm can efficiently find the optimal pricing policy for both the small and large networks, and can significantly enhance the platform profit and service efficiency.

A reinforcement learning integral sliding mode control scheme against lumped disturbances in hot strip rolling

In the hot strip rolling production process, due to the influence of external disturbances and unmodeled errors which are defined as lumped disturbances (LDs), the looper operating angle and strip tension may not satisfy the requirements of normal production under the set values, which may lead to the product not satisfying the requirements. In this paper, an advanced reinforcement learning (RL) algorithm combined with integral sliding mode control (ISMC) is utilized to deal with LDs, and then the robustness of the system is improved. Firstly, the hot strip rolling is a complex nonlinear system, the looper model is linearized to be approximated around the operating point for simplicity. Secondly, the ISMC is addressed for LDs in hot strip rolling; meanwhile, due to the chattering phenomenon in conventional sliding mode control, a novel reaching law is designed; whereafter an off-policy RL algorithm is adopted to obtain the optimal controller. Finally, the effectiveness and superiority of the addressed method are elaborated by an example.

Reinforcement Learning and Optimal Control of PMSM Speed Servo System

This article proposes a novel model-free optimal speed tracking control scheme for permanent magnet synchronous motors (PMSMs) through reinforcement learning (RL). To achieve the speed servo control, we formulate the linear quadratic regulator associated with the reduced-order model in the outer-loop controller design. Such a model is obtained in terms of singular perturbation theory, which enables the separation of slow and fast time-scale dynamics. Moreover, we develop an off-policy RL algorithm to iteratively approximate the ideal value of solution to the linear quadratic regulator without requiring any knowledge of model parameters of the PMSM and the measurement of the load torque. Both simulation and experimental tests are carried out to justify that the proposed control scheme realizes precision speed tracking performance and shape transient response in the presence of unknown model parameters.

Model-Free H∞ Output Feedback Control of Road Sensing in Vehicle Active Suspension Based on Reinforcement Learning

Abstract An active suspension system ensures the controllability of a vehicle in the vertical direction, which greatly enhances the control redundancy and safety of an intelligent driven vehicle. However, many calibrated model parameters are not conducive to the application of optimal control. To reduce the control cost of active suspension, a model-free H∞ output feedback control method is studied in this research. First, the optimal governing equation of the active suspension is transformed into a zero-sum game problem of two players, and an off-policy reinforcement learning algorithm is established to solve the game algebraic Riccati equation. This method could overcome the disadvantage of constant interactions between Q-learning and the environment. Secondly, with the consideration that some state variables are difficult to measure, a data-driven H∞ output feedback controller is designed using road sensing information and historical measurement data, and the Bellman equation of the system is solved using the least squares method to obtain the optimal control solution of the active suspension. The simulation and rapid prototype experimental results show that the proposed method could produce the optimal control strategy of the system without model parameters, overcome the strong dependence and sensitivity of traditional design methods to model parameters and improve the robust control effect of the active suspension.

Off-Policy Learning-Based Following Control of Cooperative Autonomous Vehicles Under Distributed Attacks

This paper investigates the resilient distributed secure output path following control problem of heterogeneous autonomous ground vehicles (AGVs) subject to cyber attacks based on reinforcement learning algorithm. Most existing results are subject to the same attack models for all communication channels, however multiple channels launched by different attackers are considered in this paper. First, a predictor-acknowledgement clock algorithm for each vehicle is proposed to judge whether the communication channel among neighboring vehicles is attacked or not by receiving or transmitting an acknowledgement. Then, a resilient distributed predictor is proposed to predict the pinning vehicle’s state for each vehicle. In addition, a resilient local control protocol consisting of the feedforward state provided by the predictor and the local feedback state of each vehicle is developed for the output path following problem, which is further converted to the optimal control problem by designing a discounted performance function. Discounted algebraic Riccati equations (AREs) are derived to address the optimal control problem. An off-policy reinforcement learning (RL) algorithm is put forward to learn the solution of discounted AREs online without any prior knowledge of vehicles’ dynamics. It is shown that the RL-based output path following control problem of AGVs imposed by cyber attacks can be achieved in an optimal manner. Finally, a numerical example is provided to verify the effectiveness of theoretical analysis.

Robust Output Regulation and Reinforcement Learning-Based Output Tracking Design for Unknown Linear Discrete-Time Systems

In this article, we investigate the optimal output tracking problem for linear discrete-time systems with unknown dynamics using reinforcement learning (RL) and robust output regulation theory. This output tracking problem only allows to utilize the outputs of the reference system and the controlled system, rather than their states, and differs from most existing works that depend on the state of the system. The optimal tracking problem is formulated into a linear quadratic regulation problem by proposing a family of dynamic discrete-time controllers. Then, it is shown that solving the output tracking problem is equivalent to solving output regulation equations, whose solution, however, requires the knowledge of the complete and accurate system dynamics. To remove such a requirement, an off-policy RL algorithm is proposed using only the measured output data along the trajectory of the system and the reference output. By introducing reexpression error and analyzing the rank condition of the parameterization matrix, we ensure the uniqueness of the proposed RL-based optimal control via output feedback.

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

This article studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques. Based on policy iteration, a novel off-policy reinforcement learning algorithm, named optimistic least-squares-based policy iteration, is proposed, which is able to find iteratively near-optimal policies of the adaptive optimal stationary control problem directly from input/state data without explicitly identifying any system matrices, starting from an initial admissible control policy. The solutions given by the proposed optimistic least-squares-based policy iteration are proved to converge to a small neighborhood of the optimal solution with probability one, under mild conditions. The application of the proposed algorithm to a triple inverted pendulum example validates its feasibility and effectiveness.