Reinforcement Learning Environment Research Articles

Molecule generation toward target protein (SARS-CoV-2) using reinforcement learning-based graph neural network via knowledge graph.

AI-driven approaches are widely used in drug discovery, where candidate molecules are generated and tested on a target protein for binding affinity prediction. However, generating new compounds with desirable molecular properties such as Quantitative Estimate of Drug-likeness (QED) and Dopamine Receptor D2 activity (DRD2) while adhering to distinct chemical laws is challenging. To address these challenges, we proposed a graph-based deep learning framework to generate potential therapeutic drugs targeting the SARS-CoV-2 protein. Our proposed framework consists of two modules: a novel reinforcement learning (RL)-based graph generative module with knowledge graph (KG) and a graph early fusion approach (GEFA) for binding affinity prediction. The first module uses a gated graph neural network (GGNN) model under the RL environment for generating novel molecular compounds with desired properties and a custom-made KG for molecule screening. The second module uses GEFA to predict binding affinity scores between the generated compounds and target proteins. Experiments show how fine-tuning the GGNN model under the RL environment enhances the molecules with desired properties to generate valid and unique compounds using different scoring functions. Additionally, KG-based screening reduces the search space of generated candidate molecules by while retaining of promising binding molecules against SARS-CoV-2 protein, i.e., 3C-like protease (3CLpro). We achieved a binding affinity score of 8.185 from the top rank of generated compound. In addition, we compared top-ranked generated compounds to Indinavir on different parameters, including drug-likeness and medicinal chemistry, for qualitative analysis from a drug development perspective. The online version contains supplementary material available at 10.1007/s13721-023-00409-2.

RLSF: Multimodal Sleep Improvement Based Reinforcement Learning

As informatization 3.0 accelerates the pace of people’s life and work, people’s happiness index and physical and mental health have become the focus of attention and research in sociology, psychology and medicine. Currently, neurological diseases represented by insomnia have become common chronic diseases. However, existing insomnia treatment methods mainly focus on drug therapy and EEG-based expert intervention, ignoring the individual variability of insomnia patients, the high cost of expert intervention, and the privacy of the user’s treatment environment. Therefore, aiming at the effect of white noise lite on sleep quality, this paper proposes a time-frequency domain correlation multimodal sleep enhancement framework based on reinforcement learning (RLSF), which is a closed-loop feedback sleep improvement framework that includes hardware and software. Specifically, the individual sleep state is fed for learning through EEG sensors’ input, and the agent is gradually trained to suit the sleep habit. This paper provides a reinforcement learning environment in which different agents can be deployed easily; then, we propose the Deep Net Sleep Improvement agent (DNSI agent) and the Time and Frequency-based Lightweight Sleep Improvement agent (TFLSI agent) for RLSF. Finally, the substantial experiments compare DNSI and TFLSI agent performance, and the results indicate that these two agents both have decision-making ability, and three volunteers’ Pittsburgh Sleep Quality Index significantly reduces by 3-7 points within two months and the average time to sleep is reduced by 131.4 seconds.

B-Spline-Based Curve Fitting to Cam Pitch Curve Using Reinforcement Learning

Directly applying the B-spline interpolation function to process plate cams in a computer numerical control (CNC) system may produce verbose tool-path codes and unsmooth trajectories. This paper is devoted to addressing the problem of B-spline fitting for cam pitch curves. Considering that the B-spline curve needs to meet the motion law of the follower to approximate the pitch curve, we use the radial error to quantify the effects of the fitting B-spline curve and the pitch curve. The problem thus boils down to solving a difficult global optimization problem to find the numbers and positions of the control points or data points of the B-spline curve such that the cumulative radial error between the fitting curve and the original curve is minimized, and this problem is attempted in this paper with a double deep Q-network (DDQN) reinforcement learning (RL) algorithm with data points traceability. Specifically, the RL environment, actions set and current states set are designed to facilitate the search of the data points, along with the design of the reward function and the initialization of the neural network. The experimental results show that when the angle division value of the actions set is fixed, the proposed algorithm can maximize the number of data points of the B-spline curve, and accurately place these data points to the right positions, with the minimum average of radial errors. Our work establishes the theoretical foundation for studying spline fitting using the RL method.

Optimal Manufacturing Configuration Selection: Sequential Decision Making and Optimization using Reinforcement Learning

In manufacturing, different costs must be considered when selecting the optimal manufacturing configuration. Costs include manufacturing costs, material costs, labor costs, and overhead costs. Optimal manufacturing configurations are those that minimize production criteria, such as costs, production speed, and flexibility, while still meeting the required production levels and quality standards. To find the optimal manufacturing configuration, manufacturers often use a combination of traditional techniques, e.g., mathematical modeling, simulation, and optimization, to evaluate the tradeoffs between different cost factors and identify configurations that provide the best balance between cost and performance. However, these techniques may require long development and simulation time, and/or may require expert knowledge. This paper presents a method for selecting the optimal manufacturing configuration, focusing on cost optimization, using a reinforcement learning (RL) approach for sequential decision-making. The proposed method involves developing a RL environment, requiring lower development and simulation times than traditional techniques, that captures the incurred costs, recurring costs, production rates, and setup times of manufacturing configurations. The problem is then solved using the Proximal Policy Optimization algorithm to identify the configuration that minimizes costs while still meeting the required production levels and quality standards. The effectiveness of the proposed method is validated through a machining process planning case study with multiple cost factors and production constraints. In particular, the machining process plan was developed for an industry-relevant product prototype. The results show that the proposed method can find solutions that are robust to stochastic noise, providing valuable insights for manufacturers looking to optimize manufacturing operations.

Deep Reinforcement Learning on FPGA for Self-Healing Cryogenic Power Amplifier Control

Wireless sensing and communication for space exploration in areas inaccessible to human often suffer from severe performance degradation due to the cryogenic effects on the transmitters’ circuits. To survive extreme temperatures, programmable radio frequency (RF) power amplifiers (PA) can be built into the transmitter, and intelligent PA controllers need to be integrated into the system to interact with the environment and restore the PA’s functionalities. This problem can be modeled as the controller acts (control the PA) in an environment to maximize the reward (signal quality), and it is most suitable to use reinforcement learning as a solution. This paper presents a cryogenic and energy-efficient reinforcement learning (RL) module on Field Programmable Gate Arrays (FPGA) that can directly program the PA. By characterizing a self-healing PA in a liquid nitrogen environment, we generated an RF signal data set and built an interactive RL environment to model the PA’s behaviors across its configurations and cryogenic temperatures down to −197°C. We developed a deep RL model with a high generalization capability introduced by the neural networks to control the PA and restore its performance. The RL model with fixed-point training and inference is implemented on FPGA to survive the cryogenic conditions and carry out fast and low-power training and inference for PA control. All functionalities of the programmed FPGA operate correctly in the cryogenic testing environment.

Reinforcement learning based optimization algorithm for maintenance tasks scheduling in coalbed methane gas field

The Coalbed Methane (CBM) well maintenance tasks scheduling optimization is of importance for improving the CBM production efficiency. Traditionally, this problem is addressed using mathematical model based classical optimization method or some meta-heuristic algorithms. However, due to the large-scale nature of this problem, these trials often fail in practical use. Therefore, the Q-learning algorithm based solving method is proposed in this paper. An interactive environment for reinforcement learning is constructed. To validate the effectiveness of proposed method, scenarios with different scales are provided. For the cases with 10, 14, 20, 30, 47, 60, 80 and 100 maintenance tasks respectively, the required solution time is 3.66 s, 4.94 s, 7.66 s, 12.48 s, 21.82 s, 30.82 s, 48.33 s, 73.17 s respectively. The proposed Q-learning algorithm is insensitive with the problem scale, which is promising. Moreover, the Q-learning based algorithm is more efficient than the traditional algorithm along with the increase of the number of tasks.

Robust Attitude Control of an Agile Aircraft Using Improved Q-Learning

Attitude control of a novel regional truss-braced wing (TBW) aircraft with low stability characteristics is addressed in this paper using Reinforcement Learning (RL). In recent years, RL has been increasingly employed in challenging applications, particularly, autonomous flight control. However, a significant predicament confronting discrete RL algorithms is the dimension limitation of the state-action table and difficulties in defining the elements of the RL environment. To address these issues, in this paper, a detailed mathematical model of the mentioned aircraft is first developed to shape an RL environment. Subsequently, Q-learning, the most prevalent discrete RL algorithm, will be implemented in both the Markov Decision Process (MDP) and Partially Observable Markov Decision Process (POMDP) frameworks to control the longitudinal mode of the proposed aircraft. In order to eliminate residual fluctuations that are a consequence of discrete action selection, and simultaneously track variable pitch angles, a Fuzzy Action Assignment (FAA) method is proposed to generate continuous control commands using the trained optimal Q-table. Accordingly, it will be proved that by defining a comprehensive reward function based on dynamic behavior considerations, along with observing all crucial states (equivalent to satisfying the Markov Property), the air vehicle would be capable of tracking the desired attitude in the presence of different uncertain dynamics including measurement noises, atmospheric disturbances, actuator faults, and model uncertainties where the performance of the introduced control system surpasses a well-tuned Proportional–Integral–Derivative (PID) controller.

Analog Integrated Circuit Topology Synthesis With Deep Reinforcement Learning

This article presents a novel deep-reinforcement-learning-based method for topology synthesis of analog-integrated circuits, especially operational amplifiers (OpAmps). It behaves like a human designer, who learns from trials, derives design knowledge and experience, and evolves gradually to finally figure out optimal manners to construct proper circuit topologies that meet design specifications. Essential design rules are defined and applied to set up the specialized environment for reinforcement learning in order to reasonably construct circuit topologies with building blocks as the basic components. Our proposed method can not only handle large-size circuit designs but also generate creative circuit topologies. The produced circuit topologies are verified by the simulation-in-loop sizing. In order to improve the evaluation efficiency, hash table and symbolic analysis techniques are utilized to significantly reduce the number of the produced topologies to be sized during the synthesis process. Compared with the state-of-the-art approaches, our proposed method significantly improves the synthesis efficiency by consuming only several hours on average to produce a trustworthy solution. Our experimental results demonstrate its sound efficiency, strong reliability, and wide applicability.

Distributed optimization of electricity-Gas-Heat integrated energy system with multi-agent deep reinforcement learning

The coordinated optimization problem of the electricity-gas-heat integrated energy system (IES) has the characteristics of strong coupling, non-convexity, and nonlinearity. The centralized optimization method has a high cost of communication and complex modeling. Meanwhile, the traditional numerical iterative solution cannot deal with uncertainty and solution efficiency, which is difficult to apply online. For the coordinated optimization problem of the electricity-gas- heat IES in this study, we constructed a model for the distributed IES with a dynamic distribution factor and transformed the centralized optimization problem into a distributed optimization problem in the multi-agent reinforcement learning environment using multi-agent deep deterministic policy gradient. Introducing the dynamic distribution factor allows the system to consider the impact of changes in real-time supply and demand on system optimization, dynamically coordinating different energy sources for complementary utilization and effectively improving the system economy. Compared with centralized optimization, the distributed model with multiple decision centers can achieve similar results while easing the pressure on system communication. The proposed method considers the dual uncertainty of renewable energy and load in the training. Compared with the traditional iterative solution method, it can better cope with uncertainty and realize real- time decision making of the system, which is conducive to the online application. Finally, we verify the effectiveness of the proposed method using an example of an IES coupled with three energy hub agents.

Towards Augmented Microscopy with Reinforcement Learning-Enhanced Workflows.

Here, we report a case study implementation of reinforcement learning (RL) to automate operations in the scanning transmission electron microscopy workflow. To do so, we design a virtual, prototypical RL environment to test and develop a network to autonomously align the electron beam position without prior knowledge. Using this simulator, we evaluate the impact of environment design and algorithm hyperparameters on alignment accuracy and learning convergence, showing robust convergence across a wide hyperparameter space. Additionally, we deploy a successful model on the microscope to validate the approach and demonstrate the value of designing appropriate virtual environments. Consistent with simulated results, the on-microscope RL model achieves convergence to the goal alignment after minimal training. Overall, the results highlight that by taking advantage of RL, microscope operations can be automated without the need for extensive algorithm design, taking another step toward augmenting electron microscopy with machine learning methods.

Exploring Reinforcement Learning Environment for User-centric Applications in VANET

Exploring Reinforcement Learning Environment for User-centric Applications in VANET

Assessing seismic-like events prediction in model knits with unsupervised machine learning

Knitted fabric exhibits avalanche-like events when deformed, which we are interested in predicting, by analogy with earthquakes. However, as in most analogous seismic models, the time intermittence and scale-invariance of these events severely jeopardize this endeavor. But more importantly, such predictions are hard to assess and not easily compared. Here we introduce a framework that allows not only to predict seismic-like, rebalanced time series, but to also evaluate and compare the relevance of competing predictions. It relies on a reinforcement learning environment that learns risk management in a model, seismically active city subjected to the crackling dynamics observed in the mechanical response of knitted fabric. Relying on extensive experimental data, we show that this mechanism allows to assess the finite predictability of seismic-like activity, and to compare the performance of different approaches in the operational usage of such predictions.

Asymmetric Airfoil Morphing via Deep Reinforcement Learning.

Morphing aircraft are capable of modifying their geometry configurations according to different flight conditions to improve their performance, such as by increasing the lift-to-drag ratio or reducing their fuel consumption. In this article, we focus on the airfoil morphing of wings and propose a novel morphing control method for an asymmetric deformable airfoil based on deep reinforcement learning approaches. Firstly, we develop an asymmetric airfoil shaped using piece-wise Bézier curves and modeled by shape memory alloys. Resistive heating is adopted to actuate the shape memory alloys and realize the airfoil morphing. With regard to the hysteresis characteristics exhibited in the phase transformation of shape memory alloys, we construct a second-order Markov decision process for the morphing procedure to formulate a reinforcement learning environment with hysteresis properties explicitly considered. Subsequently, we learn the morphing policy based on deep reinforcement learning techniques where the accurate information of the system model is unavailable. Lastly, we conduct simulations to demonstrate the benefits brought by our learning implementations and validate the morphing performance of the proposed method. The simulation results show that the proposed method provides an average 29.8% performance improvement over traditional methods.

Run Time Assured Reinforcement Learning for Safe Satellite Docking

Reinforcement learning promises high performance in complex tasks as well as low online storage and computation cost. However, the trial-and-error learning approach of reinforcement learning could explore unsafe behavior in the search for an optimal solution. Run time assurance (RTA) approaches can be applied to monitor behavior and ensure safety constraint satisfaction during reinforcement learning. This paper investigates the effect of RTA on reinforcement learning training performance in terms of training efficiency, safety constraint satisfaction, control efficiency, task efficiency, and training duration. For the purposes of demonstration, a custom reinforcement learning environment is created where the objective is to develop a policy that moves a satellite into docking position with another satellite in a two-dimensional relative-motion reference frame. Six different policies are trained. The first features no RTA, the second features no RTA but a higher penalty for safety violations, and four others use different RTA techniques to enforce a dynamic velocity constraint during training. The trained policies are analyzed with standardized test points. It is shown that the policies trained without RTA do not learn to adhere to the constraint, whereas all policies trained with RTA do learn to adhere to the constraint. Although more complex RTA frameworks can be better for operational use, it is found that a simple RTA framework provides the best overall results for reinforcement learning training.

E2HRL: An Energy-efficient Hardware Accelerator for Hierarchical Deep Reinforcement Learning

Recently, Reinforcement Learning (RL) has shown great performance in solving sequential decision-making and control in dynamic environment problems. Despite its achievements, deploying Deep Neural Network (DNN)-based RL is expensive in terms of time and power due to the large number of episodes required to train agents with high dimensional image representations. Additionally, at the interference the large energy footprint of deep neural networks can be a major drawback. Embedded edge devices as the main platform for deploying RL applications are intrinsically resource-constrained and deploying deep neural network-based RL on them is a challenging task. As a result, reducing the number of actions taken by the RL agent to learn desired policy, along with the energy-efficient deployment of RL, is crucial. In this article, we propose Energy Efficient Hierarchical Reinforcement Learning (E2HRL), which is a scalable hardware architecture for RL applications. E2HRL utilizes a cross-layer design methodology for achieving better energy efficiency, smaller model size, higher accuracy, and system integration at the software and hardware layers. Our proposed model for RL agent is designed based on the learning hierarchical policies, which makes the network architecture more efficient for implementation on mobile devices. We evaluated our model in three different RL environments with different level of complexity. Simulation results with our analysis illustrate that hierarchical policy learning with several levels of control improves RL agents training efficiency and the agent learns the desired policy faster compared to a non-hierarchical model. This improvement is specifically more observable as the environment or the task becomes more complex with multiple objective subgoals. We tested our model with different hyperparameters to achieve the maximum reward by the RL agent while minimizing the model size, parameters, and required number of operations. E2HRL model enables efficient deployment of RL agent on resource-constraint-embedded devices with the proposed custom hardware architecture that is scalable and fully parameterized with respect to the number of input channels, filter size, and depth. The number of processing engines (PE) in the proposed hardware can vary between 1 to 8, which provides the flexibility of tradeoff of different factors such as latency, throughput, power, and energy efficiency. By performing a systematic hardware parameter analysis and design space exploration, we implemented the most energy-efficient hardware architectures of E2HRL on Xilinx Artix-7 FPGA and NVIDIA Jetson TX2. Comparing the implementation results shows Jetson TX2 boards achieve 0.1 ∼ 1.3 GOP/S/W energy efficiency while Artix-7 FPGA achieves 1.1 ∼ 11.4 GOP/S/W, which denotes 8.8× ∼ 11× better energy efficiency of E2HRL when model is implemented on FPGA. Additionally, compared to similar works our design shows better performance and energy efficiency.

Collaborative training of heterogeneous reinforcement learning agents in environments with sparse rewards: what and when to share?

In the early stages of human life, babies develop their skills by exploring different scenarios motivated by their inherent satisfaction rather than by extrinsic rewards from the environment. This behavior, referred to as intrinsic motivation, has emerged as one solution to address the exploration challenge derived from reinforcement learning environments with sparse rewards. Diverse exploration approaches have been proposed to accelerate the learning process over single- and multi-agent problems with homogeneous agents. However, scarce studies have elaborated on collaborative learning frameworks between heterogeneous agents deployed into the same environment, but interacting with different instances of the latter without any prior knowledge. Beyond the heterogeneity, each agent’s characteristics grant access only to a subset of the full state space, which may hide different exploration strategies and optimal solutions. In this work we combine ideas from intrinsic motivation and transfer learning. Specifically, we focus on sharing parameters in actor-critic model architectures and on combining information obtained through intrinsic motivation with the aim of having a more efficient exploration and faster learning. We test our strategies through experiments performed over a modified ViZDooM’s My Way Home scenario, which is more challenging than its original version and allows evaluating the heterogeneity between agents. Our results reveal different ways in which a collaborative framework with little additional computational cost can outperform an independent learning process without knowledge sharing. Additionally, we depict the need for modulating correctly the importance between the extrinsic and intrinsic rewards to avoid undesired agent behaviors.

Comprehensive Ocean Information-Enabled AUV Path Planning Via Reinforcement Learning

The path planning of the autonomous underwater vehicle (AUV) has shown great potential in various Internet of Underwater Things (IoUT) applications. Although considerable efforts had been made, prior studies are confronted with some limitations. For one thing, existing work only uses the ocean current simulation model without introducing real ocean information, having not been supported by real data. For another, traditional path planning algorithms have strong environment dependence and lack flexibility: once the environment changes, they need to be remodeled and replanned. To overcome these challenges, this article proposes comprehensive ocean information D3QN (COID), an AUV path planning scheme exploiting comprehensive ocean information and reinforcement learning (RL), which consists of three steps. First, we introduce the comprehensive real ocean data, including weather, temperature, thermohaline, current, etc., and apply them into the regional ocean modeling system to generated reliable ocean current. Next, through well-designed state transition function and reward function, we build a 3-D grid model of ocean environment for RL. Furthermore, based on the framework of the double dueling deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> network (D3QN), COID integrates local ocean current and position features to provide state input and uses priority sampling to accelerate network convergence. The performance of COID has been evaluated and proved by numerical results, which demonstrate efficient path planning and high flexibility for expansion into different ocean environments.

Optimization of autonomous vehicle speed control mechanisms using hybrid DDPG-SHAP-DRL-stochastic algorithm

Autonomous Vehicles (AV) are the future milestones of the automobile industry, which functions without the intervention of human being. Numerous researches have been stimulated by leading automobile sectors of the world, to address the anticipated challenges in implementing the autonomous vehicles in a practical scenario. The speed control mechanism is the predominant challenge which acts in the basis of Machine Learning mechanism is the major thrust area associated with autonomous vehicles. Reinforcement Learning (RL) is the effective algorithm to solve the challenges associated with the autonomous driving of vehicles and its decision on complex scenarios. A simulative environment is advantageous for training and validation of an RL algorithm because it reduces risk and saves resources. This research work introduces a novel hybrid algorithm composed of Deep Deterministic Policy Gradient (DDPG) - SHapley Additive exPlanations (SHAP) – Deep Reinforcement Learning (DRL)-stochastic algorithm. The primary objective of this research work is to introduce an RL environment for optimizing longitudinal control.

A study on deep reinforcement learning-based crane scheduling model for uncertainty tasks

Abstract Aiming at the crane scheduling problem for uncertainty tasks in multi-crane scheduling situation, this article proposes a deep reinforcement learning-based crane scheduling modeling method that is not dependent on mathematical planning and has certain generality. First, the crane scheduling process is integrated into deep reinforcement learning framework in which the orbit space of the crane and the transportation task is environmental information and crane is the intelligent agent. Second, the interactive mode between reinforcement learning algorithm and environment is adjusted to adapt to the combined learning of multi-crane scheduling model. Last, the crane scheduling model for uncertainty tasks is constructed by optimizing reward discount factor, learning rate, and reward function intensive mode. Testing of the model is carried out based on practical crane scheduling in one steelmaking workshop. Scheduling proposal is generated and all crane tasks are completed within the planned time, which verifies the feasibility of this model. Results show that compared with manual scheduling plan, the scheduling proposal based on the new model reduces total task completion time by 11.52%, time of collision of crane routes decreases by 57.14%, and negative crane transportation distance shortens by 55.26%. The high efficiency of the scheduling model is therefore verified.

Conformer-RL: A deep reinforcement learning library for conformer generation.

Conformer‐RL is an open‐source Python package for applying deep reinforcement learning (RL) to the task of generating a diverse set of low‐energy conformations for a single molecule. The library features a simple interface to train a deep RL conformer generation model on any covalently bonded molecule or polymer, including most drug‐like molecules. Under the hood, it implements state‐of‐the‐art RL algorithms and graph neural network architectures tuned specifically for molecular structures. Conformer‐RL is also a platform for researching new algorithms and neural network architectures for conformer generation, as the library contains modular class interfaces for RL environments and agents, allowing users to easily swap components with their own implementations. Additionally, it comes with tools to visualize and save generated conformers for further analysis. Conformer‐RL is well‐tested and thoroughly documented with tutorials for each of the functionalities mentioned above, and is available on PyPi and Github: https://github.com/ZimmermanGroup/conformer-rl.