Model-free Reinforcement Learning Algorithm Research Articles

Habits and the subexploration of better transportation options: A dual-system approach

In psychology, choices are influenced by either goal-directed or habitual systems, which are represented by model-based (MB) and model-free (MF) reinforcement learning algorithms. This paper introduces these concepts into the transportation literature and shows that individuals with habitual tendencies often fail to consider new, potentially superior, alternatives. Across two experiments, participants chose between two bus options, with a third, objectively superior option introduced in a later phase. In both experiments, participants engaging in habitual behavior were less likely to explore the new alternative. Furthermore, we found habitual tendencies were moderated by self-reported stress levels, which is consistent with previous results in the psychological literature. This implies that new transportation modes or routes may remain underused as travelers adhere to habitual patterns, disregarding the introduction of potentially better alternatives, and prompt for public policies that aim to promote goal-directed behavior.

Simulation-based evaluation of model-free reinforcement learning algorithms for quadcopter attitude control and trajectory tracking

General use quadcopters have been under development for over a decade but many of their potential applications are still under evaluation and have not yet been adopted in many of the areas that could benefit from their use. While the current generation of quadcopters use a mature set of control algorithms, the next steps, especially as autonomous features are developed, should involve a more complex learning capability to be able to adapt to unknown circumstances in a safe and reliable way. This paper provides baseline quadcopter control models learnt using eight general reinforcement learning (RL) algorithms in a simulated environment, with the object of establishing a reference performance, both in terms of precision and generation cost, for a simple set of trajectories. Each algorithm uses a tailored set of hyperparameters while, additionally, the influence of random seeds is also studied. While not all algorithms converge in the allocated computing budget, the more complex ones are able to provide stable and precise control models. This paper recommends the use of the TD3 algorithm as a reference for comparison with new RL algorithms. Additional guidance for future work is provided based on the weaknesses identified in the learning process, especially regarding the strong dependence of agent performance on random seeds.

Developing an eco-driving strategy in a hybrid traffic network using reinforcement learning.

Eco-driving has garnered considerable research attention owing to its potential socio-economic impact, including enhanced public health and mitigated climate change effects through the reduction of greenhouse gas emissions. With an expectation of more autonomous vehicles (AVs) on the road, an eco-driving strategy in hybrid traffic networks encompassing AV and human-driven vehicles (HDVs) with the coordination of traffic lights is a challenging task. The challenge is partially due to the insufficient infrastructure for collecting, transmitting, and sharing real-time traffic data among vehicles, facilities, and traffic control centers, and the following decision-making of agents involved in traffic control. Additionally, the intricate nature of the existing traffic network, with its diverse array of vehicles and facilities, contributes to the challenge by hindering the development of a mathematical model for accurately characterizing the traffic network. In this study, we utilized the Simulation of Urban Mobility (SUMO) simulator to tackle the first challenge through computational analysis. To address the second challenge, we employed a model-free reinforcement learning (RL) algorithm, proximal policy optimization, to decide the actions of AV and traffic light signals in a traffic network. A novel eco-driving strategy was proposed by introducing different percentages of AV into the traffic flow and collaborating with traffic light signals using RL to control the overall speed of the vehicles, resulting in improved fuel consumption efficiency. Average rewards with different penetration rates of AV (5%, 10%, and 20% of total vehicles) were compared to the situation without any AV in the traffic flow (0% penetration rate). The 10% penetration rate of AV showed a minimum time of convergence to achieve average reward, leading to a significant reduction in fuel consumption and total delay of all vehicles.

Evaluation of the effectiveness of methodological approaches to hierarchical intelligent control of ground-air Ad-Hoc communication network

The article shows the features of intelligent management processes of a new type of ground-air communication networks that are rapidly being implemented. The dynamic nature of the conditions of operation and behavior of communication nodes, both ground and air, causes a rapid increase in the volume of service information necessary to ensure continuous and adaptive management in real time. One of the ways to solve this problem is the redistribution of management tasks at different stages of the management cycle, which is classically divided into the stages of planning, deployment and operational management. On the one hand, the increase in entropy at the planning stage complicates this process, but this approach will increase the probability of making "correct" management decisions from the standpoint of quality management (network metrics). The work investigates a new architecture of a hierarchical intelligent ground-air communication network control system based on the model-free Reinforcement learning algorithm as a Q-learning network agent and FA-OSELM online sequential extreme machine learning algorithms as node-level agents. The IСS model is presented, its adequacy is checked, and the process of its learning at the planning stage on various mobility models is shown. An important feature of the IСS training process is the application of the developed mobility model, disclosed in the article, which describes the interaction processes of communication nodes at a deeper level. The work conducted a study of the representativeness of the training sample, obtained using the developed mobility model, that was carried out relative to the existing ones, and it was determined that despite the smaller volume of the population of the initial data, it was possible to ensure a better quality of resource management.

A generation-storage coordination dispatch strategy for power system based on causal reinforcement learning

In the backdrop of global energy transformation, power systems integrating high proportions of renewable energy sources are facing unprecedented challenges in operational stability and dispatch efficiency. To address these challenges, this study introduces a generation-storage coordination real-time dispatch strategy based on Causal Power System Dynamic Reinforcement Learning (CPSDRL). Diverging from traditional reinforcement learning approaches, CPSDRL innovatively incorporates causal inference within the state prediction model – the crux of model-based reinforcement learning – thereby establishing the Power Causal Dynamic Model (PCDM). Assisted by the prior knowledge of power systems, the model significantly enhances prediction accuracy and reliability through a two-stage training process. Utilizing PCDM, this study further applies a direct policy search algorithm to optimize the real-time dispatch strategy. Experimental results indicate that the proposed method improves the stability of generation-storage coordination real-time dispatch and exhibits competitive advantages in sample efficiency and computational speed, compared to traditional model-based and model-free reinforcement learning algorithms. This method is expected to enhance the practicality and adaptability of causal reinforcement learning techniques in power system scheduling and control.

Optimal Tracking Control of Heterogeneous MASs Using Event-Driven Adaptive Observer and Reinforcement Learning.

This article considers the output tracking control problem of nonidentical linear multiagent systems (MASs) using a model-free reinforcement learning (RL) algorithm, where partial followers have no prior knowledge of the leader's information. To lower the communication and computing burden among agents, an event-driven adaptive distributed observer is proposed to predict the leader's system matrix and state, which consists of the estimated value of relative states governed by an edge-based predictor. Meanwhile, the integral input-based triggering condition is exploited to decide whether to transmit its private control input to its neighbors. Then, an RL-based state feedback controller for each agent is developed to solve the output tracking control problem, which is further converted into the optimal control problem by introducing a discounted performance function. Inhomogeneous algebraic Riccati equations (AREs) are derived to obtain the optimal solution of AREs. An off-policy RL algorithm is used to learn the solution of inhomogeneous AREs online without requiring any knowledge of the system dynamics. Rigorous analysis shows that under the proposed event-driven adaptive observer mechanism and RL algorithm, all followers are able to synchronize the leader's output asymptotically. Finally, a numerical simulation is demonstrated to verify the proposed approach in theory.

Robust hierarchical games of linear discrete-time systems based on off-policy model-free reinforcement learning

An off-policy model-free reinforcement learning (RL) algorithm is proposed for a robust hierarchical game while considering incomplete information and input constraints. The robust hierarchical game exhibits characteristics of a Stackelberg–Nash (SN) game, where equilibrium points are designated as Stackelberg–Nash–Saddle equilibrium (SNE) points. An off-policy method is employed for the RL algorithm, addressing input constraints by using excitation input instead of real-time update polices as control inputs. Moreover, a model-free method is implemented for the off-policy RL algorithm, accounting for the challenge posed by incomplete information. The goal of this paper is to develop an off-policy model-free RL algorithm to obtain approximate SNE polices of the robust hierarchical game with incomplete information and input constraints. Furthermore, the convergence and effectiveness of the off-policy model-free RL algorithm are guaranteed by proving the equivalence of Bellman equation between nominal SNE policies and approximate SNE policies. Finally, a simulation is provided to verify the advantage of the developed algorithm.

Data-Efficient Reinforcement Learning for Complex Nonlinear Systems.

This article proposes a data-efficient model-free reinforcement learning (RL) algorithm using Koopman operators for complex nonlinear systems. A high-dimensional data-driven optimal control of the nonlinear system is developed by lifting it into the linear system model. We use a data-driven model-based RL framework to derive an off-policy Bellman equation. Building upon this equation, we deduce the data-efficient RL algorithm, which does not need a Koopman-built linear system model. This algorithm preserves dynamic information while reducing the required data for optimal control learning. Numerical and theoretical analyses of the Koopman eigenfunctions for dataset truncation are discussed in the proposed model-free data-efficient RL algorithm. We validate our framework on the excitation control of the power system.

A model-free method based on RDPG for fiber diameter steady control

Optical fibers have made outstanding contributions to optical instruments such as micro-channel plates (MCPs) and micro-pore optics (MPOs). As the stricter performance requirements for the MCP and MPO, traditional diameter control methods are incapable. Additionally, the partial observation issue of fiber drawing process further limits the outcome of the existing control approaches. Innovative control algorithms are vital for the diameter precision enhancement. We investigate the significance of recurrent deterministic policy gradient (RDPG) on diameter control. RDPG consists of deep deterministic policy gradient (DDPG) and recurrent neural network (RNN) where DDPG is a model-free reinforcement learning algorithm for continuous control and RNN for addressing partial observation. The impact of RNN variants selection on control performance is further discussed. Results illustrate that the application of RNN substantially solves the partial observation and varying RNN variants play a positive role in diameter control, but bidirectional gated recurrent unit (BiGRU) reaches the best accuracy. Comparison experiment results indicate that RDPG is able to learn reasonable control strategies and achieves significantly more accurate and robust diameter control in contrast with the existing methods.

Reinforcement learning in a prisoner's dilemma

I characterize the outcomes of a class of model-free reinforcement learning algorithms, such as stateless Q-learning, in a prisoner's dilemma. The behavior is studied in the limit as players stop experimenting after sufficiently exploring their options. A closed form relationship between the learning rate and game payoffs reveals whether the players will learn to cooperate or defect. The findings have implications for algorithmic collusion and also apply to asymmetric learners with different experimentation rules.

Reinforcement Learning-Based H∞ Control of 2-D Markov Jump Roesser Systems With Optimal Disturbance Attenuation.

This article investigates model-free reinforcement learning (RL)-based H∞ control problem for discrete-time 2-D Markov jump Roesser systems ( 2 -D MJRSs) with optimal disturbance attenuation level. This is compared to existing studies on H∞ control of 2-D MJRSs with optimal disturbance attenuation levels that are off-line and use full system dynamics. We design a comprehensive model-free RL algorithm to solve optimal H∞ control policy, optimize disturbance attenuation level, and search for the initial stabilizing control policy, via online horizontal and vertical data along 2-D MJRSs trajectories. The optimal disturbance attenuation level is obtained by solving a set of linear matrix inequalities based on online measurement data. The initial stabilizing control policy is obtained via a data-driven parallel value iteration (VI) algorithm. Besides, we further certify the performance including the convergence of the RL algorithm and the asymptotic mean-square stability of the closed-loop systems. Finally, simulation results and comparisons demonstrate the effectiveness of the proposed algorithms.

A novel reinforcement learning method for the plan of generator start-up after blackout

With the expansion of the power system and the impact of natural disasters, blackouts are almost inevitable. A scientific restoration plan can effectively shorten the economic and social losses caused by blackouts. As the first step of power system restoration, the generator start-up process impacts the success rate and time consumption. Considering the diversity of constraints in the generator start-up process and the complexity of the topology, this paper proposes a planning method for the generator start-up process based on reinforcement learning. Q-learning is a model-free reinforcement learning algorithm that can cope well with complex environments and diverse constraints. First, the generator start-up problem is modeled as a Markov Decision Process, and the restoration operation and constraints are considered. Q-learning uses an artificial intelligence agent to plan the generator start-up process and guide operators. Second, different from the restoration methods of sequential restarting the generators, this paper presents a novel strategy for restoration by restoring the path of the subsequent generator while the current generator acquires cranking power. Then, the action selection mechanism is optimized by using the Upper Confidence Bound algorithm to address the shortcoming of low exploration effectiveness in the conventional Q-learning. Finally, the simulation results provide evidence of the excellent performance of the proposed method in terms of optimality and convergence.

Accelerated Reinforcement Learning via Dynamic Mode Decomposition

This work applies the decomposition principle to discrete-time reinforcement learning to solve the optimal control problems for a network of subsystems. The control design is defined as a linear quadratic regulator graphical problem, where the performance function couples subsystems' dynamics. We first present a model-free discrete-time reinforcement learning algorithm based on online behaviors without using system dynamics. This could become a prohibitively-long learning process for larger networks. To remedy this issue, we develop an efficient model-free reinforcement learning algorithm based on dynamic mode decomposition. This decomposition method reduces the size of the measured data while the dynamic information of the original network is still retained. This algorithm is then implemented online. The proposed methodology is validated using examples of a consensus network and a power system network.

Reinforcement learning algorithms: A brief survey

Reinforcement Learning (RL) is a machine learning (ML) technique to learn sequential decision-making in complex problems. RL is inspired by trial-and-error based human/animal learning. It can learn an optimal policy autonomously with knowledge obtained by continuous interaction with a stochastic dynamical environment. Problems considered virtually impossible to solve, such as learning to play video games just from pixel information, are now successfully solved using deep reinforcement learning. Without human intervention, RL agents can surpass human performance in challenging tasks. This review gives a broad overview of RL, covering its fundamental principles, essential methods, and illustrative applications. The authors aim to develop an initial reference point for researchers commencing their research work in RL. In this review, the authors cover some fundamental model-free RL algorithms and pathbreaking function approximation-based deep RL (DRL) algorithms for complex uncertain tasks with continuous action and state spaces, making RL useful in various interdisciplinary fields. This article also provides a brief review of model-based and multi-agent RL approaches. Finally, some promising research directions for RL are briefly presented.

Vehicle ride comfort optimization in the post-braking phase using residual reinforcement learning

Owing to increasing urban congestion, ensuring vehicle ride comfort during the post-braking phase has become an essential requirement. However, achieving vehicle ride comfort using current conventional methods is challenging due to the vehicles’ complex dynamics. This paper proposes a novel controller with residual reinforcement learning, combining the advantages of the model-free reinforcement learning algorithm, heuristic optimization algorithm, and prior expert knowledge to significantly improve training efficiency. The nonlinear and transient characteristics of the tire and vehicle are modeled to improve the control accuracy. On-vehicle experiments are performed using a skateboard chassis. The experimental results show that the proposed strategy achieves significant improvement in vehicle ride comfort under various braking scenarios. We believe that this technology has the potentialto alleviate vehicle discomfort issues in daily life.

An Advisor-Based Architecture for a Sample-Efficient Training of Autonomous Navigation Agents with Reinforcement Learning

Recent advancements in artificial intelligence have enabled reinforcement learning (RL) agents to exceed human-level performance in various gaming tasks. However, despite the state-of-the-art performance demonstrated by model-free RL algorithms, they suffer from high sample complexity. Hence, it is uncommon to find their applications in robotics, autonomous navigation, and self-driving, as gathering many samples is impractical in real-world hardware systems. Therefore, developing sample-efficient learning algorithms for RL agents is crucial in deploying them in real-world tasks without sacrificing performance. This paper presents an advisor-based learning algorithm, incorporating prior knowledge into the training by modifying the deep deterministic policy gradient algorithm to reduce the sample complexity. Also, we propose an effective method of employing an advisor in data collection to train autonomous navigation agents to maneuver physical platforms, minimizing the risk of collision. We analyze the performance of our methods with the support of simulation and physical experimental setups. Experiments reveal that incorporating an advisor into the training phase significantly reduces the sample complexity without compromising the agent’s performance compared to various benchmark approaches. Also, they show that the advisor’s constant involvement in the data collection process diminishes the agent’s performance, while the limited involvement makes training more effective.

Towards a model of human-cyber–physical automata and a synthesis framework for control policies

Advances in research and increasing applications of Cyber–Physical Systems (CPSs) show the need to consider factors of humans in the loop. This has led to the growing research focus on Human-Cyber–Physical Systems (HCPSs). In general, humans in an HCPS interact with both the cyber and physical systems, as well as among the humans themselves. For a better understanding, correct design, development, operation, and maintenance of HCPSs, a computational theory based on a computational model is required. This paper presents our initial work towards a model of human-cyber–physical automata (HCPA). We consider an HCPS as a combination of a human-physical system (HPS) and a CPS in which the control switches between the humans and the machines. We define an HCPA by connecting the automaton of the HPS and the automaton of the CPS through a switch control automaton. The switch control automaton makes switching decision in some critical states shared by the HPS and the CPS. Our theorem shows that the control switching between the HPS and the CPS increases the probability of satisfying a given property. We model the behaviour of a human in specified applications or even in carry out specific tasks, instead of general human intelligence. Therefore, a human can make mistakes to decision making and thus it is a probabilistic automaton with learning ability. The switching between the human and the machine is modelled by an oracle. The oracle learns about the human behaviour, the machine behaviour, as well as the environment to make the control decisions. To generate the control policies of the human and the oracle, we propose a synthesis framework to maximize the probability of the satisfaction of a property specified in Linear Temporal Logic (LTL) by the HCPA. We present a prototype implementation of the framework by extending the model-free reinforcement learning (RL) algorithm and model-free deep-RL algorithm, and our experiment shows that our synthesis framework is effective in obtaining switch policies.

Comparative study of model-based and model-free reinforcement learning control performance in HVAC systems

Reinforcement learning (RL) shows the potential to address drawbacks of rule-based control and model predictive control and exhibits great effectiveness in heating, ventilation and air conditioning (HVAC) systems. Most studies employed model-free RL to achieve building energy conservation and increase indoor comfort. However, model-free RL algorithms face the challenge of sample efficiency which causes long-time training and restricts their applications. Model-based RL is considered an alternative avenue for accelerating learning and promoting the application of RL, but it also has limitations due to the modeling approaches and accuracy. In addition, few studies propose model-based RL algorithms and investigate performance gaps between model-free and model-based RL in HVAC systems. Therefore, this study conducts a comprehensive performance comparison between model-free and model-based RL to identify the current issues with RL control in HVAC systems. The open-source building optimization testing (BOPTEST) framework is employed as the virtual environment to evaluate the control performance and computational burden. Then Dueling Deep Q-Networks and Soft Actor-Critic are developed, and a state-of-the-art model-based RL framework is employed to develop their model-based versions. The comparison results showed that all RL controllers outperform the baseline control in terms of zone temperature and operation costs. Model-based RL can achieve a control performance as good as model-free RL with a shorter training time based on its high sample efficiency. Moreover, due to massive and quickly generated data, model-based RL can accelerate the learning of RL agents, though the model is inaccurate at the early training stage. This study would provide some insights into the RL control selection and improvements in HVAC systems.

Q-learning based energy management strategy for a hybrid multi-stack fuel cell system considering degradation

The use of multi-stack fuel cells (FCs) is attracting considerable attention in electrified vehicles due to the added degrees of freedom in terms of efficiency and survivability. In a multi-stack FC hybrid electric vehicle, the power sources (FCs and the battery pack) have different energetic characteristics and their operation is influenced by the performance drifts caused by degradation. Hence, efficient power distribution for such a multi-source system is a critical issue. This paper proposes a three-layer online EMS for a recreational vehicle composed of three FCs and a battery pack. In the first layer, two online estimators are responsible for constantly updating the characteristics of each FC and the battery to be used by the power distribution algorithm. In the second layer, a rule-based method is developed to improve the calculation speed of the power distribution algorithm by deciding when it should be activated. The last layer performs the power distribution between FCs and battery using a model-free reinforcement learning (RL) algorithm called Q-learning. The proposed RL-based EMS attempts to meet the requested power while minimizing the costs of hydrogen consumption and degradation of all power sources. To justify the performance of the proposed strategy, a comprehensive benchmark with an offline EMS and two online strategies is performed under two driving cycles. In comparison with the online strategies, the proposed method based on RL reduces the defined trip cost up to 11.5 % and 13.08 % under the Real driving cycle while having a higher cost than the offline strategy by 4.78 %.

Path Planning in Dynamic Environments Based on Q-Learning

With the rapid progress of science and technology, the scope of applications for mobile robots is growing. Path planning in dynamic environment is always a challenging task for mobile robot, which shows significant impacts in the medical field and the military field. Q-learning, a model-free reinforcement learning algorithm, can recognize its surroundings and demonstrate a system making decisions for itself about how this algorithm learns to make accurate decisions about achieving optimal target. Therefore, a path planning algorithm was proposed for robots in dynamic environments based on Q-learning. This algorithm can successfully generate several viable paths in environments with both static obstacles, dynamic obstacles and target point. First, three environments with different levels of complexity were created. Afterwards, to generate several optimal paths, the path planning algorithm was conducted in multiple times in different environments. Finally, the experimental results were collected and visualized to illustrate information. The effectiveness of the proposed algorithm is validated by experiment results in solving problems of path planning in dynamic environments with required point.