Performance Of Reinforcement Learning Research Articles

SCORE: Skill-Conditioned Online Reinforcement Learning

Solving complex long-horizon tasks through Reinforcement Learning (RL) from scratch presents challenges related to efficient exploration. Two common approaches to reduce complexity and enhance exploration efficiency are (i) integrating learning-from-demonstration techniques with online RL, where the prior knowledge acquired from demonstrations is used to guide exploration, refine representations, or tailor reward functions, and (ii) using representation learning to facilitate state abstraction. In this study, we present Skill-Conditioned Online REinforcement Learning (SCORE), a novel approach that leverages these two strategies and utilizes skills acquired from an unstructured demonstrations dataset in a policy gradient RL algorithm. This integration enriches the algorithm with informative input representations, improving downstream task learning and exploration efficiency. We evaluate our method on long-horizon robotic and navigation tasks and game environments, demonstrating enhancements in online RL performance compared to the baselines. Furthermore, we show our approach’s generalization capabilities and analyze its effectiveness through an ablation study.

Optimizing a Dynamic Vehicle Routing Problem with Deep Reinforcement Learning: Analyzing State-Space Components

Background: The dynamic vehicle routing problem (DVRP) is a complex optimization problem that is crucial for applications such as last-mile delivery. Our goal is to develop an application that can make real-time decisions to maximize total performance while adapting to the dynamic nature of incoming orders. We formulate the DVRP as a vehicle routing problem where new customer requests arrive dynamically, requiring immediate acceptance or rejection decisions. Methods: This study leverages reinforcement learning (RL), a machine learning paradigm that operates via feedback-driven decisions, to tackle the DVRP. We present a detailed RL formulation and systematically investigate the impacts of various state-space components on algorithm performance. Our approach involves incrementally modifying the state space, including analyzing the impacts of individual components, applying data transformation methods, and incorporating derived features. Results: Our findings demonstrate that a carefully designed state space in the formulation of the DVRP significantly improves RL performance. Notably, incorporating derived features and selectively applying feature transformation enhanced the model’s decision-making capabilities. The combination of all enhancements led to a statistically significant improvement in the results compared with the basic state formulation. Conclusions: This research provides insights into RL modeling for DVRPs, highlighting the importance of state-space design. The proposed approach offers a flexible framework that is applicable to various variants of the DVRP, with potential for validation using real-world data.

Alleviating imbalanced problems of reinforcement learning when applying in real-time power network dispatching and control

Real-time power network dispatching and control (PDC) presents unique challenges that traditional methods cannot effectively address due to the consideration of temporal dynamic factors. Reinforcement learning (RL) has been introduced and proven to be effective. However, given the vast space for solutions, there is an urgent need to enhance the performance of RL further. Analyzing the characteristics of the power network to improve the performance of RL holds significant potential. This article presents a comprehensive analysis of power network characteristics, including imbalanced state distribution and the imbalanced action operation frequency, along with their impact on applying RL in real-time PDC to guide the algorithm design. Guided by the analysis, the paper proposes the Balance Deep Q-Network (DQN) algorithm to mitigate the negative impact of the imbalanced problems on algorithm performance. A K-means-based predefined curriculum learning (KPCL) module is proposed to address the state imbalance issue. It allows for differentiated access to operation scenarios based on their hardness, ensuring a balanced exploration of operation scenarios while maintaining diversity during training. An Option DQN module is proposed to tackle the action imbalance problem, utilizing a branching network structure and hierarchical action selection to reduce the coupling between actions with different usage frequencies, thereby enhancing the accuracy of action value estimation. The effectiveness of the Balance DQN algorithm is demonstrated through extensive experiments conducted on the Grid2Op platform with 14-bus and 36-bus cases. The results indicate that Balance DQN can efficiently alleviate the negative impacts of imbalanced problems on RL for real-time PDC agents and achieve good performance in both cases.

Deep reinforcement learning based fast charging and thermal management optimization of an electric vehicle battery pack

Existing optimal strategies for fast-charging electric vehicle batteries are predominantly at the cell level. The proposed high-fidelity methods for extending available cell models to pack level are associated with a large computational burden, making real-world implementation impossible. Further, fast charging optimization and thermal management problems are dependent. That is, the cooling system reduces battery temperature, allowing for higher charging current, while optimal current minimizes the need for cooling and, in turn, reduces thermal system power consumption. There is a lack of studies where fast charging optimization and battery thermal management problems are jointly solved. Therefore, this paper proposes a simulation study using a deep reinforcement learning (RL) approach that concurrently solves fast charging and thermal management problems for a battery pack with low computational complexity. In this regard, we formulate each cell using an electro-thermal-aging model, which accounts for the heat exchange between adjacent cells. The electro-thermal-aging model plays the role of the environment for RL and is not the focus of novelty in this work. The RL agent is then trained to output the optimal charging current and coolant mass flow rate. Moreover, the proposed methodology is examined through a numerical study where the outperformance of our model is showcased by comparing it with baseline algorithms of model predictive control (MPC) and CC–CV. Consequently, three battery packs comprising 20, 444, and 7104 cells are used, respectively. We demonstrate that RL requires less than a second to finish the simulation for 20 cells, while MPC requires more than 80 min. In addition, RL keeps the cells’ core temperature below 33 °C, but MPC results reach 40 °C. The RL performance in charging packs with 444 and 7104 cells is then compared with that of CC–CV. In terms of computation time, RL and CC–CV are nearly the same, while regarding the average core and surface temperature of the cells as well as the cell aging, RL attains better outcomes, extending the battery pack’s life by up to two years after 1000 fast charging cycles.

Compositional design of multicomponent alloys using reinforcement learning

The design of alloys has typically involved adaptive experimental synthesis and characterization guided by machine learning models fitted to available data. A bottleneck for sequential design, be it for self-driven or manual synthesis, by Bayesian Global Optimization (BGO) for example, is that the search space becomes intractable as the number of alloy elements and its compositions exceed a threshold. Here we investigate how reinforcement learning (RL) performs in the compositional design of alloys within a closed loop with manual synthesis and characterization. We demonstrate this strategy by designing a phase change multicomponent alloy (Ti27.2Ni47Hf13.8Zr12) with the highest transformation enthalpy (ΔH) -37.1 J/g (-39.0 J/g with further calibration) within the TiNi-based family of alloys from a space of over 2 × 108 candidates, although the initial training is only on a compact dataset of 112 alloys. We show how the training efficiency is increased by employing acquisition functions containing uncertainties, such as expected improvement (EI), as the reward itself. Existing alloy data is often limited, however, if the agent is pretrained on experimental results prior to the training process, it can access regions of higher reward values more frequently. In addition, the experimental feedback enables the agent to gradually explore new regions with higher rewards, compositionally different from the initial dataset. Our approach directly applies to processing conditions where the actions would be performed in a given order. We also compare RL performance to BGO and the genetic algorithm on several test functions to gain insight on their relative strengths in materials design.

Transferable multi-objective factory layout planning using simulation-based deep reinforcement learning

Factory layout planning aims at finding an optimized layout configuration under consideration of varying influences such as the material flow characteristics. Manual layout planning can be characterized as a complex decision-making process due to a large number of possible placement options. Automated planning approaches aim at reducing the manual planning effort by generating optimized layout variants in the early stages of layout planning. Recent developments have introduced deep Reinforcement Learning (RL) based planning approaches that allow to optimize a layout under consideration of a single optimization criterion. However, within layout planning, multiple partially conflicting planning objectives have to be considered. Such multiple objectives are not considered by existing RL-based approaches. This paper addresses this research gap by presenting a novel deep RL-based layout planning approach that allows consideration of multiple objectives for optimization. Furthermore, existing RL-based planning approaches only consider analytically formulated objectives such as the transportation distance. Consequently, dynamic influences in the material flow are neglected which can result in higher operational costs of the future factory. To address this issue, a discrete event simulation module is developed that allows simulating manufacturing and material flow processes simultaneously for any layout configuration generated by the RL approach. Consequently, the presented approach considers material flow simulation results for multi-objective optimization. To investigate the capabilities of RL-based factory layout planning, different RL architectures are compared based on a simplified application scenario. Throughput time, media supply, and material flow clarity are considered as optimization objectives. The best performing architecture is then applied to an exemplary application scenario and compared with the results obtained by a combined version of the genetic algorithm and tabu search, the non-dominated sorting genetic algorithm, and the optimal solution. Finally, two industrial planning scenarios, one focusing on brownfield and one on greenfield planning, are considered. The results show that the performance of RL compared to meta-heuristics depends on the considered computation time. With time the results generated by the RL approach exceed the quality of the best conventional solution by up to 11%. Finally, the potential of applying transfer learning is investigated for three different application scenarios. It is observed that RL can learn generalized patterns for factory layout planning, which allows to significantly reduce the required training time and can lead to improved solution quality. Thus, the use of pre-trained RL models shows a substantial performance potential for automated factory layout planning which cannot be achieved with conventional automated planning approaches.

Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to learn a policy using only pre-collected and fixed data. Although avoiding the time-consuming online interactions in RL, it poses challenges for out-of-distribution (OOD) state actions and often suffers from data inefficiency for training. Despite many efforts being devoted to addressing OOD state actions, the latter (data inefficiency) receives little attention in offline RL. To address this, this paper proposes the cross-domain offline RL, which assumes offline data incorporate additional source-domain data from varying transition dynamics (environments), and expects it to contribute to the offline data efficiency. To do so, we identify a new challenge of OOD transition dynamics, beyond the common OOD state actions issue, when utilizing cross-domain offline data. Then, we propose our method BOSA, which employs two support-constrained objectives to address the above OOD issues. Through extensive experiments in the cross-domain offline RL setting, we demonstrate BOSA can greatly improve offline data efficiency: using only 10% of the target data, BOSA could achieve 74.4% of the SOTA offline RL performance that uses 100% of the target data. Additionally, we also show BOSA can be effortlessly plugged into model-based offline RL and noising data augmentation techniques (used for generating source-domain data), which naturally avoids the potential dynamics mismatch between target-domain data and newly generated source-domain data.

Solving Partially Observable 3D-Visual Tasks with Visual Radial Basis Function Network and Proximal Policy Optimization

Visual Reinforcement Learning (RL) has been largely investigated in recent decades. Existing approaches are often composed of multiple networks requiring massive computational power to solve partially observable tasks from high-dimensional data such as images. Using State Representation Learning (SRL) has been shown to improve the performance of visual RL by reducing the high-dimensional data into compact representation, but still often relies on deep networks and on the environment. In contrast, we propose a lighter, more generic method to extract sparse and localized features from raw images without training. We achieve this using a Visual Radial Basis Function Network (VRBFN), which offers significant practical advantages, including efficient and accurate training with minimal complexity due to its two linear layers. For real-world applications, its scalability and resilience to noise are essential, as real sensors are subject to change and noise. Unlike CNNs, which may require extensive retraining, this network might only need minor fine-tuning. We test the efficiency of the VRBFN representation to solve different RL tasks using Proximal Policy Optimization (PPO). We present a large study and comparison of our extraction methods with five classical visual RL and SRL approaches on five different first-person partially observable scenarios. We show that this approach presents appealing features such as sparsity and robustness to noise and that the obtained results when training RL agents are better than other tested methods on four of the five proposed scenarios.

Human-Guided Reinforcement Learning With Sim-to-Real Transfer for Autonomous Navigation.

Reinforcement learning (RL) is a promising approach in unmanned ground vehicles (UGVs) applications, but limited computing resource makes it challenging to deploy a well-behaved RL strategy with sophisticated neural networks. Meanwhile, the training of RL on navigation tasks is difficult, which requires a carefully-designed reward function and a large number of interactions, yet RL navigation can still fail due to many corner cases. This shows the limited intelligence of current RL methods, thereby prompting us to rethink combining RL with human intelligence. In this paper, a human-guided RL framework is proposed to improve RL performance both during learning in the simulator and deployment in the real world. The framework allows humans to intervene in RL's control progress and provide demonstrations as needed, thereby improving RL's capabilities. An innovative human-guided RL algorithm is proposed that utilizes a series of mechanisms to improve the effectiveness of human guidance, including human-guided learning objective, prioritized human experience replay, and human intervention-based reward shaping. Our RL method is trained in simulation and then transferred to the real world, and we develop a denoised representation for domain adaptation to mitigate the simulation-to-real gap. Our method is validated through simulations and real-world experiments to navigate UGVs in diverse and dynamic environments based only on tiny neural networks and image inputs. Our method performs better in goal-reaching and safety than existing learning- and model-based navigation approaches and is robust to changes in input features and ego kinetics. Furthermore, our method allows small-scale human demonstrations to be used to improve the trained RL agent and learn expected behaviors online.

Reinforcement Learning versus Model Predictive Control on greenhouse climate control

The greenhouse system plays a crucial role to ensure an adequate supply of fresh food for the growing global population. However, maintaining an optimal growing climate within a greenhouse requires resources and operational costs. To achieve economical and sustainable crop growth, efficient climate control in greenhouse production is paramount. Model Predictive Control (MPC) and Reinforcement Learning (RL) are the two approaches representing model-based and learning-based control, respectively. Each one has its own way to formulate control problems, define control objectives, and seek for optimal control actions that provide sustainable crop growth. Although certain forms of MPC and RL have been applied to greenhouse climate control, limited research has comprehensively analyzed the connections, differences, advantages, and disadvantages between these two approaches, both mathematically and in terms of performance. Therefore, this paper aims to address this gap by: (1) introducing a novel RL approach that utilizes Deep Deterministic Policy Gradient (DDPG) for large and continuous state–action space environments; (2) formulating the MPC and RL approaches for greenhouse climate control within a unified framework; (3) exploring the mathematical connections and differences between MPC and RL; (4) conducting a simulation study to analyze and compare the performance of MPC and RL; (5) presenting and interpreting the comparative results to provide valuable insights for the application of these control approaches in different scenarios. By undertaking these objectives, this paper seeks to contribute to the understanding and advancement of both MPC and RL methods in greenhouse climate control, fostering more informed decision-making regarding their selection and implementation based on specific requirements and constraints.

Reinforcement learning in robotic motion planning by combined experience-based planning and self-imitation learning

High-quality and representative data is essential for both Imitation Learning (IL)- and Reinforcement Learning (RL)-based motion planning tasks. For real robots, it is challenging to collect enough qualified data either as demonstrations for IL or experiences for RL due to safety consideration in environments with obstacles. We target this challenge by proposing the self-imitation learning by planning plus (SILP+) algorithm, which efficiently embeds experience-based planning into the learning architecture to mitigate the data-collection problem. The planner generates demonstrations based on successfully visited states from the current RL policy, and the policy improves by learning from these demonstrations. In this way, we relieve the demand for human expert operators to collect demonstrations required by IL and improve the RL performance as well. Various experimental results shows that SILP+ achieves better training efficiency, higher and more stable success rate in complex motion planning tasks compared to several other methods. Extensive tests on physical robots illustrate the effectiveness of SILP+ in a physical setting, retaining a success rate of 90% where the next-best contender drops from 87% to 75% in the Sim2Real transition.

Curiosity-tuned experience replay for wargaming decision modeling without reward-engineering

Reinforcement Learning (RL) has become a promising technique to deal with the tough decision modeling problem in the wargaming field. However, to deploy current RL algorithms requires reward-engineering scenario by scenario, which is laborious for massive wargaming scenarios. To tackle this issue, this paper proposes an improved RL method, curiosity-tuned experience replay (CTER), which allows the RL-driven decision model to achieve a relatively effective policy under the sparse reward. CTER uses the curiosity mechanism to regulate the three critical procedures during learning with experience replay: the exploration, storage, and revisitation of the experiences. Based on the prediction-based curiosity, CTER generates an intrinsic reward to fill the sparse reward space, and further provides an adaptive exploration strategy to collect more informative experiences. Moreover, CTER develops a novel prioritized replay and memory updating mechanism to reuse experiences more efficiently. Through the systematic evaluation and comparison on typical game tasks and wargaming tasks, CTER shows its effectiveness and generalization in different scenarios without reward-engineering. Especially, the policy performance of CTER-based RL with the sparse reward is almost equivalent to that of ordinary RL with dense engineered rewards. Our work may offer a relatively universal approach for wargaming decision modeling, which can free the RL-based decision modelers from the laborious reward-engineering.

Asynchronous Deep Double Dueling Q-learning for trading-signal execution in limit order book markets.

We employ deep reinforcement learning (RL) to train an agent to successfully translate a high-frequency trading signal into a trading strategy that places individual limit orders. Based on the ABIDES limit order book simulator, we build a reinforcement learning OpenAI gym environment and utilize it to simulate a realistic trading environment for NASDAQ equities based on historic order book messages. To train a trading agent that learns to maximize its trading return in this environment, we use Deep Dueling Double Q-learning with the APEX (asynchronous prioritized experience replay) architecture. The agent observes the current limit order book state, its recent history, and a short-term directional forecast. To investigate the performance of RL for adaptive trading independently from a concrete forecasting algorithm, we study the performance of our approach utilizing synthetic alpha signals obtained by perturbing forward-looking returns with varying levels of noise. Here, we find that the RL agent learns an effective trading strategy for inventory management and order placing that outperforms a heuristic benchmark trading strategy having access to the same signal.

Improvement of Reinforcement Learning With Supermodularity.

Reinforcement learning (RL) is a promising approach to tackling learning and decision-making problems in a dynamic environment. Most studies on RL focus on the improvement of state evaluation or action evaluation. In this article, we investigate how to reduce action space by using supermodularity. We consider the decision tasks in the multistage decision process as a collection of parameterized optimization problems, where state parameters dynamically vary along with the time or stage. The optimal solutions of these parameterized optimization problems correspond to the optimal actions in RL. For a given Markov decision process (MDP) with supermodularity, the monotonicity of the optimal action set and the optimal selection with respect to state parameters can be obtained by using the monotone comparative statics. Accordingly, we propose a monotonicity cut to remove unpromising actions from the action space. Taking bin packing problem (BPP) as an example, we show how the supermodularity and monotonicity cut work in RL. Finally, we evaluate the monotonicity cut on the benchmark datasets reported in the literature and compare the proposed RL with some popular baseline algorithms. The results show that the monotonicity cut can effectively improve the performance of RL.

Impact of network settings on reinforcement learning based caching policy in cooperative edge networks

Reinforcement learning (RL) has been used in combination with cooperative caching to deal with growing traffic in mobile networks, but the performance of RL based caching policies depends heavily on network settings. This paper investigates the impact of access delays within network infrastructures and popularity and similarity properties of the contents requested on network performance. A deep Q-network based caching framework is established in both basic and extended cooperative edge networks. Our simulation results reveal explicit relationships between the performance and influential parameters, which can provide a guidance and benchmark for the design of effective caching polices with RL and cooperation technologies.

강화학습 기반 화학 공정 제어 성능 향상을 위한 보상 함수 시뮬레이션 연구

Process control is vital for operating chemical process safely and efficiently while maintaining product quality. Process control has been implemented by utilizing traditional techniques such as PID (Proportional-Integrate-Derivative) control and MPC (Model Predictive Control). However, these approaches have disadvantages when the PID tuning rules do not provide optimal tuning parameters for various operating scenarios or when the model fails to match the actual plant. RL (Reinforcement Learning) is one of the most prominent data-driven techniques for addressing such issues and has gained popularity in recent years. While RL has been extensively studied and succeeds in controlling chemical processes, the reward function has a significant impact on its performance. Therefore, it is essential to specify the reward function in order to efficiently control the process. In this study, we propose a reward function to enhance the performance of RL and compare the control performances of MPC and RLs with four distinct reward functions. The controllers track the setpoint of the product in the Van de Vusse process and are evaluated based on the deviation between the setpoint and output. RL with the proposed reward function outperformed the other controllers. Its performance was 12% greater than that of MPC and 6-30% greater than other RL controllers. In chemical processes, the control performance of RL is enhanced by incorporating the time term and positive reward in the reward function, thus outperforming conventional control approaches.

Switching to online: Testing the validity of supervised remote testing for online reinforcement learning experiments.

Online experiments are an alternative for researchers interested in conducting behavioral research outside the laboratory. However, an online assessment might become a challenge when long and complex experiments need to be conducted in a specific order or with supervision from a researcher. The aim of this study was to test the computational validity and the feasibility of a remote and synchronous reinforcement learning (RL) experiment conducted during the social-distancing measures imposed by the pandemic. An additional feature of this study was to describe how a behavioral experiment originally created to be conducted in-person was transformed into an online supervised remote experiment. Open-source software was used to collect data, conduct statistical analysis, and do computational modeling. Python codes were created to replicate computational models that simulate the effect of working memory (WM) load over RL performance. Our behavioral results indicated that we were able to replicate remotely and with a modified behavioral task the effects of working memory (WM) load over RL performance observed in previous studies with in-person assessments. Our computational analyses using Python code also captured the effects of WM load over RL as expected, which suggests that the algorithms and optimization methods were reliable in their ability to reproduce behavior. The behavioral and computational validation shown in this study and the detailed description of the supervised remote testing may be useful for researchers interested in conducting long and complex experiments online.Supplementary InformationThe online version contains supplementary material available at 10.3758/s13428-022-01982-6.

Medium-term Capacity Management through Reinforcement Learning – Literature review and concept for an industrial pilot-application

Empty storage shelves and long customer lead times due to a sharp rise in market demand from industries on the one hand (e.g., pharmaceutical products). On the other hand, increasing short-term working or unemployment due to a rapid decline in demand (e.g., automotive). Current supply and demand gaps caused by the COVID-19 pandemic remind us that successful competition in volatile business environments requires rapid adjustments of production capacities.Capacity management (CM) addresses these adjustments by adapting production capacity to market demand. Operations managers of flexible manufacturing systems can adjust the capacity by using various levers (e.g., overtime, used machines, …). To guide these managers, decision support systems (DSS) exist for short-term CM (e.g., shop floor scheduling). However, due to complexity and runtime problems, the decision-making process for medium-term CM is usually carried out with low technical support.Increases in computing power and advances in algorithm performance over the past decades have enabled Machine Learning to solve ever more complex problems such as the aforementioned issues. Reinforcement Learning (RL) in particular has shown good performance in solving short-term CM problems when compared to humans or other established heuristics.In this work we review the current literature for CM using RL in flexible manufacturing systems. We identify an existing lack of knowledge within the overlap of medium-term CM and RL. However, good performance of RL in short-term CM indicates that an application in medium-term CM should be evaluated. In addition, we propose a concept of a method for medium-term CM based on RL to support operations managers in the decision-making process. The resulting DSS could have a significant impact on production performance, especially in terms of capacity adjustment speed.

Federated reinforcement learning: techniques, applications, and open challenges

This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e. Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

Predicting Attention-Shaping Response in People With Schizophrenia.

People with schizophrenia often experience attentional impairments that hinder learning during psychological interventions. Attention shaping is a behavioral technique that improves attentiveness in this population. Because reinforcement learning (RL) is thought to be the mechanism by which attention shaping operates, we investigated if preshaping RL performance predicted level of response to attention shaping in people with schizophrenia. Contrary to hypotheses, a steeper attentiveness growth curve was predicted by less intact pretreatment RL ability and lower baseline attentiveness, accounting for 59% of the variance. Moreover, baseline attentiveness accounted for over 13 times more variance in response to attention shaping than did RL ability. Results suggest attention shaping is most effective for lower-functioning patients, and those high in RL ability may already be close to ceiling in terms of their response to reinforcers. Attention shaping may not be a primarily RL-driven intervention, and other mechanisms of its effects should be considered.