Reinforcement Learning For Optimization Research Articles

Observation Time Effects in Reinforcement Learning on Contracts for Difference

In this paper, we present a study on Reinforcement Learning optimization models for automatic trading, in which we focus on the effects of varying the observation time. Our Reinforcement Learning agents feature a Convolutional Neural Network (CNN) together with Long Short-Term Memory (LSTM) and act on the basis of different observation time spans. Each agent tries to maximize trading profit by buying or selling one of a number of contracts in a simulated market environment for Contracts for Difference (CfD), considering correlations between individual assets by architecture. To decide which action to take on a specific contract, an agent develops a policy which relies on an observation of the whole market for a certain period of time. We investigate whether or not there exists an optimal observation sequence length, and conclude that such a value depends on market dynamics.

Reinforcement Learning Power Control Algorithm Based on Graph Signal Processing for Ultra-Dense Mobile Networks

Ultra-dense mobile networks (UDMNs) represent a promising technology for improving the network performance and providing the ubiquitous network accessibility in the beyond 5 G (B5G) mobile networks. Heterogenous densely deployed networks can dynamically offer high spectrum efficiency and enhance frequency reuse, which ultimately improves quality of service (QoS) and the user experience. However, mass inter- or intra-cell interference generated from overlap between small cells greatly limits network performance, especially when there is mobility between UEs and access points (APs). Even so, when network density increases, the complexity of conventional allocation methods can increase also. In this paper, we investigate a power control of downlink (DL) connection in the UNMNs with different types of APs. We propose a reinforcement learning (RL) power allocation algorithm based on graph signal processing (GSP) for ultra-dense mobile networks. Firstly, we construct a realistic system model under ultra-dense mobile networking, which includes the system channel mode and instantaneous rate. Then we employ a GSP tool to analyze network interference, the interference analysis results for the entire network are obtained to determine optimal RL power allocation. Finally, simulation results indicate that the proposed RL power control algorithm outperforms baseline algorithms when applied to a ultra-dense mobile networks.

Deep reinforcement learning optimization framework for a power generation plant considering performance and environmental issues

In the electric power generation sector, striking a balance between maximum power production and acceptable emission limits is a challenging task that requires sophisticated techniques. With traditional methods, this is an extremely complex issue due to the large number of process variables that are involved. In this paper, a deep reinforcement learning optimization framework (DRLOF) is proposed to determine the optimal operating conditions for a commercial circulating fluidized bed (CFB) power plant that strikes a good balance between performance and environmental issues. The DRLOF included the CFB as an environment created from over 1.5 years of plant data with a 1 min sampling time which interacted with an advantage actor-critic (A2C) agent of two architectures named ‘separate-A2CN’ and ‘shared-A2CN’. The framework was optimized by maximizing electric power generation within the constraints of the plant’s capacity and environmental emission standards, taking into consideration the cost of operations. After training, the framework of the separate-A2CN architecture achieved a 1.97% increase in electricity generation and 1.59% emission reduction for NOx at 14.3 times lower computational cost. Furthermore, we demonstrated the framework’s flexibility, adaptability and lower computational burden by carrying out different test scenarios which demonstrated the effectiveness of the DRLOF. The findings of this study are not limited to the CFB power plant but can be extended to other chemical processes and industries. This approach minimizes the need for costly experiments, online optimization challenges and associated customizations. • Multi-objective optimization of CFB power plant with deep reinforcement learning. • Objective formulation considered power, fuel, reagent and environmental standards. • Two main reinforcement learning architectures were analysed for better CFB results. • Improved performance of CFB by 1.97% power increase and 1.59% emission reduction. • The framework’s generality, adaptability and computational efficiency were tested.

A reinforcement learning optimization for future smart cities using software defined networking

Nowadays smart cities towards software defined network (SDN) approach will become better flexibility and manageability. A stronger, more dynamic network is an SDN network, which is precisely what a smart city network must be if it wants to be viable on a real-world scale. SDN architecture is developed to implement a learning framework for network optimization. The proposed method is called mixed-integer and reinforcement learned network optimization (MI-RLNO) for SDN monitoring. In the first phase, mixed-integer programming formulation is used as an optimization formulation for latency and convergence time. In the second phase, a reinforced Q Learning model is designed that uses communication and computation time as input state vector. Optimization formulation is used as the actions and strategies to be followed during the design and operation of communication networks, therefore contributing fairness and throughput. Simulation results improved the efficiency of the MI-RLNO method.

Constrained Q-Learning for Batch Process Optimization

Chemical process optimization and control often require satisfaction of constraints for safe operation. Reinforcement learning (RL) has been shown to be a powerful control technique that can handle nonlinear stochastic optimal control problems. Despite this promise, RL has yet to see significant translation to industrial practice due to its inability to satisfy state constraints. This work aims to address this challenge. We propose an “oracle”-assisted constrained Q-learning algorithm that guarantees the satisfaction of joint chance constraints with high probability, which is required for safety critical tasks. To that end, constraint tightening (backoffs) are introduced, which are adjusted using Broyden’s method, hence making the backoffs self-tuned. This results in a general methodology that can be integrated into approximate dynamic programming-based algorithms to guarantee constraint satisfaction with high probability. Finally, a case study is presented to compare the performance of the proposed approach with that of model predictive control (MPC). The superior performance of the proposed algorithm, in terms of constraint handling, signifies a step toward the use of RL in real world optimization and control of systems, where constraints are critical in ensuring safety.

Optimization of Drilling Cost Using Artificial Intelligence

Drilling operations in the oil and gas industry takes most of the well cost and how fast the drilling bit penetrate and bore the formation is termed the Rate of penetration (ROP). Since most of the cost incurred during drilling is related to the drilling operations, there is need not only to drill carefully, but also to optimize the drilling process. A lot of parameters are related to the rate of penetration which are actually interdependent on each other. This makes it difficult to predict the influence of every single parameter Drilling optimization techniques have been used recently to reduce drilling operation costs. There are different approaches to optimizing the cost of drilling oil and gas wells, some of which include static and /or real time optimization of drilling parameters. A potential area for optimization of drilling cost is through bit run in the well but this is particularly difficult due to its significance in both drilling time and bit cost. In this sense, as a particular bit gets used, it gets dull as its footage increases, resulting from the reduction in the bit penetration rate. The reduction in penetration rate increases total drill time. In order to optimize bit cost, it is desirable to find a trade-off between the two by a bit change policy This study is aimed at minimizing drilling time by use of artificial intelligent for the bit program. Data obtained from a well in the Niger delta region of Nigeria was used in this study and the cost optimization modelled as a Markov decision process where the intelligent agent was to learn the optimal timings for bit change by reinforcement policy Iteration learning. This study was able to achieve its objectives as the reinforcement learning optimization process performed very well with time as the computer agent was able to figure out how to improve drilling cost over time. Better results could be obtained with a better hardware and increased training time.

Physics-informed reinforcement learning optimization of nuclear assembly design

Optimization of nuclear fuel assemblies if performed effectively, will lead to fuel efficiency improvement, cost reduction, and safety assurance. However, assembly optimization involves solving high-dimensional and computationally expensive combinatorial problems. As such, fuel designers’ expert judgement has commonly prevailed over the use of stochastic optimization (SO) algorithms such as genetic algorithms and simulated annealing. To improve the state-of-art, we explore a class of artificial intelligence (AI) algorithms, namely, reinforcement learning (RL) in this work. We propose a physics-informed AI optimization methodology by establishing a connection through reward shaping between RL and the tactics fuel designers follow in practice by moving fuel rods in the assembly to meet specific constraints and objectives. The methodology utilizes RL algorithms, deep Q learning and proximal policy optimization, and compares their performance to SO algorithms. The methodology is applied on two boiling water reactor assemblies of low-dimensional (∼2×106 combinations) and high-dimensional (∼1031 combinations) natures. The results demonstrate that RL is more effective than SO in solving high dimensional problems, i.e., 10 × 10 assembly, through embedding expert knowledge in form of game rules and effectively exploring the search space. For a given computational resources and timeframe relevant to fuel designers, RL algorithms outperformed SO through finding more feasible patterns, 4–5 times more than SO, and through increasing search speed, as indicated by the RL outstanding computational efficiency. The results of this work clearly demonstrate RL effectiveness as another decision support tool for nuclear fuel assembly optimization.

Learning Automata-Based Multiagent Reinforcement Learning for Optimization of Cooperative Tasks.

Multiagent reinforcement learning (MARL) has been extensively used in many applications for its tractable implementation and task distribution. Learning automata, which can be classified under MARL in the category of independent learner, are used to obtain the optimal joint action or some type of equilibrium. Learning automata have the following advantages. First, learning automata do not require any agent to observe the action of any other agent. Second, learning automata are simple in structure and easy to be implemented. Learning automata have been applied to function optimization, image processing, data clustering, recommender systems, and wireless sensor networks. However, a few learning automata-based algorithms have been proposed for optimization of cooperative repeated games and stochastic games. We propose an algorithm known as learning automata for optimization of cooperative agents (LA-OCA). To make learning automata applicable to cooperative tasks, we transform the environment to a P-model by introducing an indicator variable whose value is one when the maximal reward is obtained and is zero otherwise. Theoretical analysis shows that all the strict optimal joint actions are stable critical points of the model of LA-OCA in cooperative repeated games with an arbitrary finite number of players and actions. Simulation results show that LA-OCA obtains the pure optimal joint strategy with a success rate of 100% in all of the three cooperative tasks and outperforms the other algorithms in terms of learning speed.

On Entropy Regularized Path Integral Control for Trajectory Optimization.

In this article, we present a generalized view on Path Integral Control (PIC) methods. PIC refers to a particular class of policy search methods that are closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems. This class is unique in the sense that it can be solved explicitly yielding a formal optimal state trajectory distribution. In this contribution, we first review the PIC theory and discuss related algorithms tailored to policy search in general. We are able to identify a generic design strategy that relies on the existence of an optimal state trajectory distribution and finds a parametric policy by minimizing the cross-entropy between the optimal and a state trajectory distribution parametrized by a parametric stochastic policy. Inspired by this observation, we then aim to formulate a SOC problem that shares traits with the LSOC setting yet that covers a less restrictive class of problem formulations. We refer to this SOC problem as Entropy Regularized Trajectory Optimization. The problem is closely related to the Entropy Regularized Stochastic Optimal Control setting which is often addressed lately by the Reinforcement Learning (RL) community. We analyze the theoretical convergence behavior of the theoretical state trajectory distribution sequence and draw connections with stochastic search methods tailored to classic optimization problems. Finally we derive explicit updates and compare the implied Entropy Regularized PIC with earlier work in the context of both PIC and RL for derivative-free trajectory optimization.

Enzyme activities predicted by metabolite concentrations and solvent capacity in the cell.

Experimental measurements or computational model predictions of the post-translational regulation of enzymes needed in a metabolic pathway is a difficult problem. Consequently, regulation is mostly known only for well-studied reactions of central metabolism in various model organisms. In this study, we use two approaches to predict enzyme regulation policies and investigate the hypothesis that regulation is driven by the need to maintain the solvent capacity in the cell. The first predictive method uses a statistical thermodynamics and metabolic control theory framework while the second method is performed using a hybrid optimization–reinforcement learning approach. Efficient regulation schemes were learned from experimental data that either agree with theoretical calculations or result in a higher cell fitness using maximum useful work as a metric. As previously hypothesized, regulation is herein shown to control the concentrations of both immediate and downstream product concentrations at physiological levels. Model predictions provide the following two novel general principles: (1) the regulation itself causes the reactions to be much further from equilibrium instead of the common assumption that highly non-equilibrium reactions are the targets for regulation; and (2) the minimal regulation needed to maintain metabolite levels at physiological concentrations maximizes the free energy dissipation rate instead of preserving a specific energy charge. The resulting energy dissipation rate is an emergent property of regulation which may be represented by a high value of the adenylate energy charge. In addition, the predictions demonstrate that the amount of regulation needed can be minimized if it is applied at the beginning or branch point of a pathway, in agreement with common notions. The approach is demonstrated for three pathways in the central metabolism of E. coli (gluconeogenesis, glycolysis-tricarboxylic acid (TCA) and pentose phosphate-TCA) that each require different regulation schemes. It is shown quantitatively that hexokinase, glucose 6-phosphate dehydrogenase and glyceraldehyde phosphate dehydrogenase, all branch points of pathways, play the largest roles in regulating central metabolism.

WSN-Assisted UAV Trajectory Adjustment for Pesticide Drift Control

Unmanned Aerial Vehicles (UAVs) have been widely applied for pesticide spraying as they have high efficiency and operational flexibility. However, the pesticide droplet drift caused by wind may decrease the pesticide spraying efficiency and pollute the environment. A precision spraying system based on an airborne meteorological monitoring platform on manned agricultural aircrafts is not adaptable for. So far, there is no better solution for controlling droplet drift outside the target area caused by wind, especially by wind gusts. In this regard, a UAV trajectory adjustment system based on Wireless Sensor Network (WSN) for pesticide drift control was proposed in this research. By collecting data from ground WSN, the UAV utilizes the wind speed and wind direction as inputs to autonomously adjust its trajectory for keeping droplet deposition in the target spraying area. Two optimized algorithms, namely deep reinforcement learning and particle swarm optimization, were applied to generate the newly modified flight route. At the same time, a simplified pesticide droplet drift model that includes wind speed and wind direction as parameters was developed and adopted to simulate and compute the drift distance of pesticide droplets. Moreover, an LSTM-based wind speed prediction model and a RNN-based wind direction prediction model were established, so as to address the problem of missing the latest wind data caused by communication latency or a lack of connection with the ground nodes. Finally, experiments were carried out to test the communication latency between UAV and ground WSN, and to evaluate the proposed scheme with embedded Raspberry Pi boards in UAV for feasibility verification. Results show that the WSN-assisted UAV trajectory adjustment system is capable of providing a better performance of on-target droplet deposition for real time pesticide spraying with UAV.

A hierarchical constrained reinforcement learning for optimization of bitumen recovery rate in a primary separation vessel

This work proposes a two-level hierarchical constrained control structure for reinforcement learning (RL) with application in a Primary Separation Vessel (PSV). The lower level is concerned with servo tracking and regulation of the interface level against variances in ore quality by manipulating middlings flow rate. At the higher level, with the objective to optimize bitumen recovery rate, a supervisory interface level setpoint control is implemented. To prevent sanding, tailings density regulation using tailings withdrawal flow rate is proposed. For each case, an asynchronous advantage actor-critic (A3C) based agent is chosen to interact with a high-fidelity PSV model to learn the near optimal control strategy through episodic interactions. Each of the three control loops is sequentially learnt. In the interface level control loop, a behavioral cloning based two-phase learning scheme to promote stable state space exploration is proposed. The proposed hierarchical structure successfully demonstrates improved bitumen recovery rate by manipulating the interface level while preventing sanding.

BOLeRo: Behavior optimization and learning for robots

Reinforcement learning and behavior optimization are becoming more and more popular in the field of robotics because algorithms are mature enough to tackle real problems in this domain. Robust implementations of state-of-the-art algorithms are often not publicly available though, and experiments are hardly reproducible because open-source implementations are often not available or are still in a stage of research code. Consequently, often it is infeasible to deploy these algorithms on robotic systems. BOLeRo closes this gap for policy search and evolutionary algorithms by delivering open-source implementations of behavior learning algorithms for robots. It is easy to integrate in robotic middlewares and it can be used to compare methods and develop prototypes in simulation.

Online optimal and adaptive integral tracking control for varying discrete‐time systems using reinforcement learning

SummaryConventional closed‐form solution to the optimal control problem using optimal control theory is only available under the assumption that there are known system dynamics/models described as differential equations. Without such models, reinforcement learning (RL) as a candidate technique has been successfully applied to iteratively solve the optimal control problem for unknown or varying systems. For the optimal tracking control problem, existing RL techniques in the literature assume either the use of a predetermined feedforward input for the tracking control, restrictive assumptions on the reference model dynamics, or discounted tracking costs. Furthermore, by using discounted tracking costs, zero steady‐state error cannot be guaranteed by the existing RL methods. This article therefore presents an optimal online RL tracking control framework for discrete‐time (DT) systems, which does not impose any restrictive assumptions of the existing methods and equally guarantees zero steady‐state tracking error. This is achieved by augmenting the original system dynamics with the integral of the error between the reference inputs and the tracked outputs for use in the online RL framework. It is further shown that the resulting value function for the DT linear quadratic tracker using the augmented formulation with integral control is also quadratic. This enables the development of Bellman equations, which use only the system measurements to solve the corresponding DT algebraic Riccati equation and obtain the optimal tracking control inputs online. Two RL strategies are thereafter proposed based on both the value function approximation and the Q‐learning along with bounds on excitation for the convergence of the parameter estimates. Simulation case studies show the effectiveness of the proposed approach.

A novel axle temperature forecasting method based on decomposition, reinforcement learning optimization and neural network

Axle temperature forecasting technology is important for monitoring the status of the train bogie and preventing the hot axle and other dangerous accidents. In order to achieve high-precision forecasting of axle temperature, a hybrid axle temperature time series forecasting model based on decomposition preprocessing method, parameter optimization method, and the Back Propagation (BP) neural network is proposed in this study. The modeling process consists of three phases. In stage I, the empirical wavelet transform (EWT) method is used to preprocess the original axle temperature series by decomposing them into several subseries. In stage II, the Q-learning algorithm is used to optimize the initial weights and thresholds of the BP neural network. In stage III, the Q-BPNN network is used to build the forecasting model and complete predicting all subseries. And the final forecasting results are generated by combining all prediction results of subseries. By comparing all results over three case predictions, it can be concluded that: (a) the proposed Q-learning based parameter optimization method is effective in improving the accuracy of the BP neural network and works better than the traditional population-based optimization methods; (b) the proposed hybrid axle temperature forecasting model can get accurate prediction results in all cases and provides the best accuracy among eight general models.

Optimization for Reinforcement Learning: From a single agent to cooperative agents

This article reviews recent advances in multi-agent reinforcement learning algorithms for large-scale control systems and communication networks, which learn to communicate and cooperate. We provide an overview of this emerging field, with an emphasis on the decentralized setting under different coordination protocols. We highlight the evolution of reinforcement learning algorithms from single-agent to multi-agent systems, from a distributed optimization perspective, and conclude with future directions and challenges, in the hope to catalyze the growing synergy among distributed optimization, signal processing, and reinforcement learning communities.

UCRLF: unified constrained reinforcement learning framework for phase-aware architectures for autonomous vehicle signaling and trajectory optimization

Signaling and trajectory optimization work as contention and researchers have debated on what should be the best for the vehicle, but it seems that both components are complement to each other and there can be combined situations with bounds where maximum optimization can be achieved. This paper introduces a novel approach called Phase-Aware Deep Learning and Constrained Reinforcement Learning for optimization and constant improvement of signal and trajectory for autonomous vehicle operation modules for an intersection. It deals with all the components required for the signaling system to operate, communicate and also navigate the vehicle with proper trajectory so that it faces less waiting time and the overall system operates with minimum waiting time and comparable throughput rate. We have done analysis on the operating time and the vehicle movement as these are vital for pollution and energy consumption. Our methodologies are not only efficient in time and computation but also have incorporated highly optimized data representation to reduce the overhead of maintaining and accessing the data. This ensures very efficient time complexity and theoretical computation time and better lower bounds. Constrained Reinforcement Learning concept is the main contribution of this work and it helped in decreasing 84% of the waiting time for the vehicles.

Distributed Reinforcement Learning Algorithm for Dynamic Economic Dispatch With Unknown Generation Cost Functions

In this article, the dynamic economic dispatch (DED) problem for smart grid is solved under the assumption that no knowledge of the mathematical formulation of the actual generation cost functions is available. The objective of the DED problem is to find the optimal power output of each unit at each time so as to minimize the total generation cost. To address the lack of a priori knowledge, a new distributed reinforcement learning optimization algorithm is proposed. The algorithm combines the state-action-value function approximation with a distributed optimization based on multiplier splitting. Theoretical analysis of the proposed algorithm is provided to prove the feasibility of the algorithm, and several case studies are presented to demonstrate its effectiveness.

Directed Exploration in Black-Box Optimization for Multi-Objective Reinforcement Learning

Usually, real-world problems involve the optimization of multiple, possibly conflicting, objectives. These problems may be addressed by Multi-objective Reinforcement learning (MORL) techniques. MORL is a generalization of standard Reinforcement Learning (RL) where the single reward signal is extended to multiple signals, in particular, one for each objective. MORL is the process of learning policies that optimize multiple objectives simultaneously. In these problems, the use of directional/gradient information can be useful to guide the exploration to better and better behaviors. However, traditional policy-gradient approaches have two main drawbacks: they require the use of a batch of episodes to properly estimate the gradient information (reducing in this way the learning speed), and they use stochastic policies which could have a disastrous impact on the safety of the learning system. In this paper, we present a novel population-based MORL algorithm for problems in which the underlying objectives are reasonably smooth. It presents two main characteristics: fast computation of the gradient information for each objective through the use of neighboring solutions, and the use of this information to carry out a geometric partition of the search space and thus direct the exploration to promising areas. Finally, the algorithm is evaluated and compared to policy gradient MORL algorithms on different multi-objective problems: the water reservoir and the biped walking problem (the latter both on simulation and on a real robot).

A Novel Medical Image Edge Detection Method Based on Reinforcement Learning and Ant Colony Optimization

A Novel Medical Image Edge Detection Method Based on Reinforcement Learning and Ant Colony Optimization