Model-free Reinforcement Learning Research Articles

Reinforcement Learning for Fuselage Shape Control during Aircraft Assembly

Critical safety requirements necessitate ultra-high precision quality control during the assembly of large aerospace components to reduce the mismatch between parts to be joined. Traditional methods use heuristic shape adjustment or surrogate model-based control. These methods are limited by reliance on accurate model learning and inadequate robustness to varying initial assembly conditions. To address these limitations, this paper proposes a model-free reinforcement learning approach for adaptive fuselage shape control during aircraft assembly. The trained reinforcement learning agent directly adjusts the aircraft components in response to their part variations and enables an autonomous system (like AlphaGo) to learn the optimal shape control policy. Specifically, the reinforcement learning environment is built on the finite element simulator. A reward function is developed to capture the optimization objective and introduces a scheme to enforce the original constraints. The proximal policy optimization algorithm is modified to speed up the learning progress and achieve better final performance. In the case study, the root-mean-square gap between components is reduced by 98.4% on average compared with their initial shape mismatch. Our proposed method outperforms the benchmark methods with smaller final shape errors, smaller maximum forces, and lower variations across different test samples.

Dyna-PINN: Physics-informed deep dyna-q reinforcement learning for intelligent control of building heating system in low-diversity training data regimes

This paper introduces Dyna-PINN, a novel physics-informed Deep Dyna-Q (DDQ) reinforcement learning (RL) approach, designed to address the data-intensive training requirements and model-agnostic nature of the conventional model-free RL methods. The DDQ approach blends model-based and model-free elements to enhance both learning and decision-making processes. By utilizing a physics-informed neural network (PINN) based model, our method enriches the learning process with physical information, enhancing the agent's planning capabilities and leading to faster learning compared to conventional model-free RL methods like Deep Q-Network (DQN) in scenarios with low-diversity training data availability. Our results demonstrate that Dyna-PINN has 50% greater sample efficiency than DQN and outperforms rule-based control in terms of thermal discomfort. Due to physics incorporation, the Dyna-PINN implements a more logical and interpretable control policy. It shows consistently good performance compared to all control variants across low-diversity data scenarios, i.e., 6 weeks of building data, and in higher-diversity data regimes, i.e., 6 months of building energy data, demonstrating the value of physics incorporation into the RL training. Additionally, we present two other DDQ-based techniques, RC-DDQ and NN-DDQ, exploring the synergy between neural networks and physical data in intelligent control designs for building energy systems. Rigorous controller testing is performed using the Building Optimization and Testing Framework (BOPTEST), a high-fidelity simulator that closely represents a real building's operation. Through comprehensive comparisons and realistic simulations, our study underscores the effectiveness of incorporating physics-informed approaches into RL-based control strategies, paving the way for more efficient and robust building energy management systems.

An Overview Of Deep Deterministic Policy Gradient Algorithm And Applications

This research summarizes the major developments of the Deep Deterministic Policy Gradient (DDPG) in reinforcement learning Motivated by the ideas, of Deep Q-networks, DDPG has proven capable of confronting more complex problems involving continuous action spaces. The core of DDPG lies in its actor-critic architecture, which enables the learning of highly competitive policies. By leveraging neural network function approximations, it can efficiently operate in large state and action spaces. DDPG has found practical applications across various real-world domains. However, like many model-free reinforcement learning methods, DDPG still faces the challenge of requiring a large number of training steps.

AGENT Q-LEARNING SPEED DEPENDENCY ON THE FACTORS IN TIC-TAC-TOE

Reinforcement learning of program agents has widespread usage today. Especially, Q-learning, a model-free reinforcement learning technique has shown great results in various applications, like games, self-driving car and robot control. Regarding turn-based games many scientists successfully applied it to train artificial intelligence and to make competitive opponent for human player. While algorithms are well-known there is a room for parametrical optimization to achieve maximum learning speed considering specific problem like turn-based board game. As results of this research show the speed can vary significantly. Tic-Tac-Toe is a very simple and old game that gives an opportunity to try Q-learning without excessive efforts. The algorithm is universal and can be applied to more complex games. It worth noting that core of learning algorithm is the same for any similar game – only rules and board size are changed, which is one of the important properties of Q-learning. This paper investigates the impact of learning rate and discount factor on the speed of learning of the Q-learning program agent in Tic-Tac-Toe board game. It is conducted a series of experiments using developed computer implementation of the algorithm to analyze the correlation between learning rate, discount factor, and the convergence rate of Q-learning in the specified game. So the experimental field consists of two factors, each of which has three levels, and full factorial experiment has nine combinations of these factors. The learning speed dependency on the each factor is presented. The findings reveal strong relationships between these parameters and convergence speed. For example, the speed is increased proportionally to the both factors, but in the case of discount factor the increase is about 1.4 times lower. Practical significance of the research is in the optimization of factors to achieve effective training of the software agent in order to save processing time, the payment of which is one of the main expenses of enterprises in the field of information technologies. In addition, the research contributes to a better understanding of how Q-learning performs in different game scenarios and provides guidelines for parameter selection in similar applications

Vision-based reinforcement learning control of soft robot manipulators

PurposeThis study aims to tackle control challenges in soft robots by proposing a visually-guided reinforcement learning approach. Precise tip trajectory tracking is achieved for a soft arm manipulator.Design/methodology/approachA closed-loop control strategy uses deep learning-powered perception and model-free reinforcement learning. Visual feedback detects the arm’s tip while efficient policy search is conducted via interactive sample collection.FindingsPhysical experiments demonstrate a soft arm successfully transporting objects by learning coordinated actuation policies guided by visual observations, without analytical models.Research limitations/implicationsConstraints potentially include simulator gaps and dynamical variations. Future work will focus on enhancing adaptation capabilities.Practical implicationsBy eliminating assumptions on precise analytical models or instrumentation requirements, the proposed data-driven framework offers a practical solution for real-world control challenges in soft systems.Originality/valueThis research provides an effective methodology integrating robust machine perception and learning for intelligent autonomous control of soft robots with complex morphologies.

Physical model-assisted deep reinforcement learning for energy management optimization of industrial electric-hydrogen coupling system with hybrid energy storage

Utilizing renewable energy sources (RESs), such as wind and solar, to convert electrical energy into hydrogen energy for industrial users with different types of energy storage can enhance the integration of green electricity, thereby replacing fossil fuels like natural gas. In the energy management optimization of industrial electric‑hydrogen coupling system (EHCS) with high RESs integration, the traditional optimization methods cannot effectively address the nonlinear characteristics of equipment and stochastic RES generation, while the model-free reinforcement learning methods struggle to improve the training efficiency and may lead to constraint violation. To address this challenge, this paper proposes a short time scale energy management approach for EHCS based on physical models assisted deep reinforcement learning (DRL). Firstly, the energy management optimization model of the EHCS is developed based on the operation characteristics of the equipment within the system. Secondly, the energy management model of EHCS is formulated as a DRL framework. The DRL agent is efficiently trained with the assistance of the action transformation which takes into account the physical characteristics of the equipment during operation. Finally, a case study is conducted to verify the effectiveness of the proposed model for achieving high-efficiency agent training, reducing decision-making time for equipment scheduling, addressing the stochastic RES generation, and realizing the economic operation of EHCS with good performance based on the equipment characteristics.

Deep reinforcement learning control of combined chemotherapy and anti-angiogenic drug delivery for cancerous tumor treatment

By virtue of the chronic and dangerous nature of cancer, researchers have explored various approaches to managing the abnormal cell growth associated with this disease using novel treatment methods. This study introduces a control system based on normalized advantage function reinforcement learning. It aims to boost the body's immune response against cancer cell proliferation. This control approach is applied to provide a combination of both chemotherapy and anti-angiogenic drugs for the first time without the need for complex, predefined mathematical models. It employs a model-free reinforcement learning technique that adaptively adjusts to individual patients to determine optimal drug administration with minimum injection rates. In this regard, a comprehensive and realistic simulation and training environment is employed, with the concentrations of normal cells, cancer cells, and endothelial cells, as well as the levels of chemotherapy and anti-angiogenic agents, as state variables. Furthermore, high levels of disturbances are considered in the simulation to investigate the robustness of the proposed method against probable uncertainties in the treatment process or patient parameters. A practical reward function has also been devised in alignment with medical objectives to ensure effective and safe treatment outcomes. The results demonstrate robustness and superior performance compared to the existing methods. Simulations show that the proposed approach is a dependable strategy for effectively reducing the concentration of cancer cells in the shortest duration using minimal doses of chemotherapy and anti-angiogenic drugs.

Habits and the subexploration of better transportation options: A dual-system approach

In psychology, choices are influenced by either goal-directed or habitual systems, which are represented by model-based (MB) and model-free (MF) reinforcement learning algorithms. This paper introduces these concepts into the transportation literature and shows that individuals with habitual tendencies often fail to consider new, potentially superior, alternatives. Across two experiments, participants chose between two bus options, with a third, objectively superior option introduced in a later phase. In both experiments, participants engaging in habitual behavior were less likely to explore the new alternative. Furthermore, we found habitual tendencies were moderated by self-reported stress levels, which is consistent with previous results in the psychological literature. This implies that new transportation modes or routes may remain underused as travelers adhere to habitual patterns, disregarding the introduction of potentially better alternatives, and prompt for public policies that aim to promote goal-directed behavior.

Reinforcement Twinning: From digital twins to model-based reinforcement learning

The concept of digital twins promises to revolutionize engineering by offering new avenues for optimization, control, and predictive maintenance. We propose a novel framework for simultaneously training the digital twin of an engineering system and an associated control agent. The training of the twin combines methods from adjoint-based data assimilation and system identification, while the training of the control agent combines model-based optimal control and model-free reinforcement learning. The training of the control agent is achieved by letting it evolve independently along two paths: one driven by a model-based optimal control and another driven by reinforcement learning. The virtual environment offered by the digital twin is used as a playground for confrontation and indirect interaction. This interaction occurs as an “expert demonstrator”, where the best policy is selected for the interaction with the real environment and “cloned” to the other if the independent training stagnates. We refer to this framework as Reinforcement Twinning (RT). The framework is tested on three vastly different engineering systems and control tasks, namely (1) the control of a wind turbine subject to time-varying wind speed, (2) the trajectory control of flapping-wing micro air vehicles (FWMAVs) subject to wind gusts, and (3) the mitigation of thermal loads in the management of cryogenic storage tanks. The test cases are implemented using simplified models for which the ground truth on the closure law is available. The results show that the adjoint-based training of the digital twin is remarkably sample-efficient and completed within a few iterations. Concerning the control agent training, the results show that the model-based and the model-free control training benefit from the learning experience and the complementary learning approach of each other. The encouraging results open the path towards implementing the RT framework on real systems.

Simulation-based evaluation of model-free reinforcement learning algorithms for quadcopter attitude control and trajectory tracking

General use quadcopters have been under development for over a decade but many of their potential applications are still under evaluation and have not yet been adopted in many of the areas that could benefit from their use. While the current generation of quadcopters use a mature set of control algorithms, the next steps, especially as autonomous features are developed, should involve a more complex learning capability to be able to adapt to unknown circumstances in a safe and reliable way. This paper provides baseline quadcopter control models learnt using eight general reinforcement learning (RL) algorithms in a simulated environment, with the object of establishing a reference performance, both in terms of precision and generation cost, for a simple set of trajectories. Each algorithm uses a tailored set of hyperparameters while, additionally, the influence of random seeds is also studied. While not all algorithms converge in the allocated computing budget, the more complex ones are able to provide stable and precise control models. This paper recommends the use of the TD3 algorithm as a reference for comparison with new RL algorithms. Additional guidance for future work is provided based on the weaknesses identified in the learning process, especially regarding the strong dependence of agent performance on random seeds.

MFRL-BI: Design of a model-free reinforcement learning process control scheme by using Bayesian inference

Design of the process control scheme is critical for quality assurance to reduce variations in manufacturing systems. Taking semiconductor manufacturing as an example, extensive literature focuses on control optimization based on certain process models (usually linear models), which are obtained by experiments before a manufacturing process starts. However, in real applications, pre-defined models may not be accurate, especially for a complex manufacturing system. To tackle model inaccuracy, we propose a Model-Free Reinforcement Learning (MFRL) approach to conduct experiments and optimize control simultaneously according to real-time data. Specifically, we design a novel MFRL control scheme by updating the distribution of disturbances using Bayesian inference to reduce their large variations during manufacturing processes. As a result, the proposed MFRL controller is demonstrated to perform well in a nonlinear chemical mechanical planarization process when the process model is unknown. Theoretical properties are guaranteed when disturbances are additive. The numerical studies also demonstrate the efficiency of our methodology.

A safe reinforcement learning algorithm for supervisory control of power plants

Traditional control theory-based methods require tailored engineering for each system and constant fine-tuning. In power plant control, one often needs to obtain a precise representation of the system dynamics and carefully design the control scheme accordingly. Model-free Reinforcement learning (RL) has emerged as a promising solution for control tasks due to its ability to learn from trial-and-error interactions with the environment. It eliminates the need for explicitly modeling the environment’s dynamics, which is potentially inaccurate. However, the direct imposition of state constraints in power plant control raises challenges for standard RL methods. To address this, we propose a chance-constrained RL algorithm based on Proximal Policy Optimization for supervisory control. Our method employs Lagrangian relaxation to convert the constrained optimization problem into an unconstrained objective, where trainable Lagrange multipliers enforce the state constraints. Our approach achieves the smallest distance of violation and violation rate in a load-follow maneuver for an advanced Nuclear Power Plant design.

Learning optimal edge processing with offloading and energy harvesting

Modern portable devices can execute increasingly sophisticated AI models on sensed data. The complexity of such processing tasks is data-dependent and has relevant energy cost. This work develops an Age of Information Markovian model for a system where multiple battery-operated devices perform data processing and energy harvesting in parallel. Part of their computational burden is offloaded to an edge server which polls devices at given rate. The structural properties of an optimal policy for a single device-server system are derived. They permit to define a new model-free reinforcement learning method specialized for monotone policies, namely Ordered Q-Learning, providing a fast procedure to learn the optimal policy. The method is oblivious to the devices’ battery capacities, the cost and the value of data batch processing and to the dynamics of the energy harvesting process. Finally, the polling strategy of the server is optimized by combining this policy improvement technique with stochastic approximation methods. Extensive numerical results provide insight into the system properties and demonstrate that the proposed learning algorithms outperform existing baselines.

Stocks and Options Portfolio Optimisation With Reinforcement Learning

We present a model-free reinforcement learning (RL) framework for portfolio optimization across multiple assets and option prices. We directly model the relationship between the portfolio weights and the predictors with a network and maximize performance metrics for portfolio construction by using RL. We construct a portfolio, such that for any quantity of option that we buy or sell, we have a quasi-replicating portfolio made of a quantity of stock and a quantity of cash. We propose Cross-Asset-Option Transformers (CAOTs) to recover the interrelationships among options and their corresponding assets, which we use to construct a long-short or a bottom-up portfolio. We then maximize performance metrics of our portfolio with RL.

Developing an eco-driving strategy in a hybrid traffic network using reinforcement learning.

Eco-driving has garnered considerable research attention owing to its potential socio-economic impact, including enhanced public health and mitigated climate change effects through the reduction of greenhouse gas emissions. With an expectation of more autonomous vehicles (AVs) on the road, an eco-driving strategy in hybrid traffic networks encompassing AV and human-driven vehicles (HDVs) with the coordination of traffic lights is a challenging task. The challenge is partially due to the insufficient infrastructure for collecting, transmitting, and sharing real-time traffic data among vehicles, facilities, and traffic control centers, and the following decision-making of agents involved in traffic control. Additionally, the intricate nature of the existing traffic network, with its diverse array of vehicles and facilities, contributes to the challenge by hindering the development of a mathematical model for accurately characterizing the traffic network. In this study, we utilized the Simulation of Urban Mobility (SUMO) simulator to tackle the first challenge through computational analysis. To address the second challenge, we employed a model-free reinforcement learning (RL) algorithm, proximal policy optimization, to decide the actions of AV and traffic light signals in a traffic network. A novel eco-driving strategy was proposed by introducing different percentages of AV into the traffic flow and collaborating with traffic light signals using RL to control the overall speed of the vehicles, resulting in improved fuel consumption efficiency. Average rewards with different penetration rates of AV (5%, 10%, and 20% of total vehicles) were compared to the situation without any AV in the traffic flow (0% penetration rate). The 10% penetration rate of AV showed a minimum time of convergence to achieve average reward, leading to a significant reduction in fuel consumption and total delay of all vehicles.

Optimization of latching control for duck wave energy converter based on deep reinforcement learning

In the field of wave energy extraction, employing active control strategies amplifies the Wave Energy Converter's (WEC) response to wave motion. In this regard, a numerical simulation study was conducted on the interaction between Duck WEC and waves under discrete latching control, proposing an optimized latching duration strategy based on Deep Reinforcement Learning (DRL). The study utilized an improved Edinburgh Duck model to establish a numerical wave flume (NWF), verifying the motion of waves and the device. A DRL and Computational Fluid Dynamics (CFD) coupled framework was developed to implement the latching control strategy. In the limited trichromatic wave testing conditions provided in this paper, the latching control time is optimized using a DRL agent, and its capture efficiency is compared with two non-predictive latching control strategies. The DRL agent control showed a 13.4% improvement relative to reference threshold control, with less than 6.3% differences compared to optimal threshold latching results. Improvement of approximately 5.6% was observed compared to optimal constant latching control. For the first time in nonlinear fluid dynamics numerical simulation, this study demonstrates the efficacy of model-free reinforcement learning with discrete latching actions, establishing an environment-driven control strategy, and having the potential to become a general strategy for WEC latching control.

A goal-conditioned policy search method with multi-timescale value function tuning

PurposeCurrent reinforcement learning (RL) algorithms are facing issues such as low learning efficiency and poor generalization performance, which significantly limit their practical application in real robots. This paper aims to adopt a hybrid model-based and model-free policy search method with multi-timescale value function tuning, aiming to allow robots to learn complex motion planning skills in multi-goal and multi-constraint environments with a few interactions.Design/methodology/approachA goal-conditioned model-based and model-free search method with multi-timescale value function tuning is proposed in this paper. First, the authors construct a multi-goal, multi-constrained policy optimization approach that fuses model-based policy optimization with goal-conditioned, model-free learning. Soft constraints on states and controls are applied to ensure fast and stable policy iteration. Second, an uncertainty-aware multi-timescale value function learning method is proposed, which constructs a multi-timescale value function network and adaptively chooses the value function planning timescales according to the value prediction uncertainty. It implicitly reduces the value representation complexity and improves the generalization performance of the policy.FindingsThe algorithm enables physical robots to learn generalized skills in real-world environments through a handful of trials. The simulation and experimental results show that the algorithm outperforms other relevant model-based and model-free RL algorithms.Originality/valueThis paper combines goal-conditioned RL and the model predictive path integral method into a unified model-based policy search framework, which improves the learning efficiency and policy optimality of motor skill learning in multi-goal and multi-constrained environments. An uncertainty-aware multi-timescale value function learning and selection method is proposed to overcome long horizon problems, improve optimal policy resolution and therefore enhance the generalization ability of goal-conditioned RL.

Mitigating Value Hallucination in Dyna-Style Planning via Multistep Predecessor Models

Dyna-style reinforcement learning (RL) agents improve sample efficiency over model-free RL agents by updating the value function with simulated experience generated by an environment model. However, it is often difficult to learn accurate models of environment dynamics, and even small errors may result in failure of Dyna agents. In this paper, we highlight that one potential cause of that failure is bootstrapping off of the values of simulated states, and introduce a new Dyna algorithm to avoid this failure. We discuss a design space of Dyna algorithms, based on using successor or predecessor models---simulating forwards or backwards---and using one-step or multi-step updates. Three of the variants have been explored, but surprisingly the fourth variant has not: using predecessor models with multi-step updates. We present the \emph{Hallucinated Value Hypothesis} (HVH): updating the values of real states towards values of simulated states can result in misleading action values which adversely affect the control policy. We discuss and evaluate all four variants of Dyna amongst which three update real states toward simulated states --- so potentially toward hallucinated values --- and our proposed approach, which does not. The experimental results provide evidence for the HVH, and suggest that using predecessor models with multi-step updates is a fruitful direction toward developing Dyna algorithms that are more robust to model error.

Evaluation of the effectiveness of methodological approaches to hierarchical intelligent control of ground-air Ad-Hoc communication network

The article shows the features of intelligent management processes of a new type of ground-air communication networks that are rapidly being implemented. The dynamic nature of the conditions of operation and behavior of communication nodes, both ground and air, causes a rapid increase in the volume of service information necessary to ensure continuous and adaptive management in real time. One of the ways to solve this problem is the redistribution of management tasks at different stages of the management cycle, which is classically divided into the stages of planning, deployment and operational management. On the one hand, the increase in entropy at the planning stage complicates this process, but this approach will increase the probability of making "correct" management decisions from the standpoint of quality management (network metrics). The work investigates a new architecture of a hierarchical intelligent ground-air communication network control system based on the model-free Reinforcement learning algorithm as a Q-learning network agent and FA-OSELM online sequential extreme machine learning algorithms as node-level agents. The IСS model is presented, its adequacy is checked, and the process of its learning at the planning stage on various mobility models is shown. An important feature of the IСS training process is the application of the developed mobility model, disclosed in the article, which describes the interaction processes of communication nodes at a deeper level. The work conducted a study of the representativeness of the training sample, obtained using the developed mobility model, that was carried out relative to the existing ones, and it was determined that despite the smaller volume of the population of the initial data, it was possible to ensure a better quality of resource management.

POMDP inference and robust solution via deep reinforcement learning: an application to railway optimal maintenance

Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the unavailability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), typically benefit from the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of Transformers and long short-term memory networks, which constitute model-free RL solutions and work directly on the observation space, with an approach termed the belief-input method, which works on the belief space by exploiting the learned POMDP model for belief inference. We apply these methods to the real-world problem of optimal maintenance planning for railway assets and compare the results with the current real-life policy. We show that the RL policy learned by the belief-input method is able to outperform the real-life policy by yielding significantly reduced life-cycle costs.