Conventional Reinforcement Learning Algorithms Research Articles

Learning with sparse reward in a gap junction network inspired by the insect mushroom body.

Animals can learn in real-life scenarios where rewards are often only available when a goal is achieved. This 'distal' or 'sparse' reward problem remains a challenge for conventional reinforcement learning algorithms. Here we investigate an algorithm for learning in such scenarios, inspired by the possibility that axo-axonal gap junction connections, observed in neural circuits with parallel fibres such as the insect mushroom body, could form a resistive network. In such a network, an active node represents the task state, connections between nodes represent state transitions and their connection to actions, and current flow to a target state can guide decision making. Building on evidence that gap junction weights are adaptive, we propose that experience of a task can modulate the connections to form a graph encoding the task structure. We demonstrate that the approach can be used for efficient reinforcement learning under sparse rewards, and discuss whether it is plausible as an account of the insect mushroom body.

Empty container repositioning problem using a reinforcement learning framework with multi-weight adaptive reward function

ABSTRACT In logistics networks, empty container congestion and scarcity often stem from trade imbalance and supply-demand mismatch. This paper focuses on the problem of empty container repositioning in maritime logistics and proposes a reinforcement learning framework that integrates a self-adaptive mechanism for adjusting the weights of a multi-objective reward function. The objective is to enhance container utilization and reduce scheduling costs. By reviewing the development of the empty container repositioning problem and analysing the advantages of using reinforcement learning to address the temporal and spatial complexity, the problem is modelled as a Markov decision process and tackled using reinforcement learning techniques. To achieve the optimization objectives, which involve reducing resource shortages at various locations and minimizing resource repositioning costs, a multi-objective reward function is introduced to capture the mutually constrained preferences. The weights of the reward function are dynamically adjusted to account for the potential time-varying preferences of the agent, mitigating the issue of poor generalization performance associated with fixed-weight reward functions. Comparative experimental analysis against conventional reinforcement learning algorithms demonstrates the superior performance of the proposed approach in problem solving. Based on the results and practical requirements of the case study, relevant recommendations for addressing empty container repositioning are presented.

Deep Reinforcement Learning-Based Security-Constrained Battery Scheduling in Home Energy System

The home energy system today involves multiple renewable energy sources and battery energy storage systems, which can be considered as a microgrid. The battery energy storage system is a key component in the home energy system for the sake of filling the gap between the user demand and volatile energy supplies to maximize the techno and economic performances. However, the battery scheduling must suffer the stochastic nature of renewable energy resources and loads, which results in an intractable multi-period stochastic optimization problem with security constraints. An improved actor-critic-based reinforcement learning is proposed for this issue, where a distributional critic net is applied for faster and more accurate reward estimation under uncertainties, and a policy net incorporating protective secondary control is adopted to satisfy security constraints, preventing the unsafe state of batteries during the trial-and-error process. Numerical tests show that the proposed approach outperforms conventional reinforcement learning algorithms, as well as the rule-based battery scheduling approach while guaranteeing safe operation. The robustness and adaptability of the proposed method are also verified in case studies with different optimization tasks.

Distributional Policy Gradient With Distributional Value Function.

In this article, we propose a distributional policy-gradient method based on distributional reinforcement learning (RL) and policy gradient. Conventional RL algorithms typically estimate the expectation of return, given a state-action pair. Furthermore, distributional RL algorithms consider the return as a random variable and estimate the return distribution that can characterize the probability of different returns resulted by environmental uncertainties. Thus, the return distribution provides more valuable information than its expectation, leading to superior policies in general. Although distributional RL has been investigated widely in value-based RL methods, very few policy-gradient methods take advantage of distributional RL. To bridge this research gap, we propose a distributional policy-gradient method by introducing a distributional value function to the policy gradient (DVDPG). We estimate the distribution of policy gradient instead of the expectation estimated in conventional policy-gradient RL methods. Furthermore, we propose two policy-gradient value sampling mechanisms to do policy improvement. First, we propose a distribution-probability-sampling method that samples the policy-gradient value according to the quantile probability of return distribution. Second, a uniform sample mechanism is proposed. With our sample mechanisms, the proposed distributional policy-gradient method enhances the stochasticity of the policy gradient, improving the exploration efficiency and benefiting to avoid falling into local optimal solutions. In sparse-reward tasks, the distribution-probability-sampling method outperforms the uniform sample mechanism. In dense-reward tasks, the two sample mechanisms perform similarly. Moreover, we show that the conventional policy-gradient method is a special case of the proposed method. Experimental results on various sparse-reward and dense-reward OpenAI-gym tasks illustrate the efficiency of the proposed method, outperforming baselines in almost environments.

Dynamic resource management in integrated NOMA terrestrial–satellite networks using multi-agent reinforcement learning

The integration of terrestrial and satellite wireless communication networks offers a practical solution to enhance network coverage, connectivity, and cost-effectiveness. Moreover, in today’s interconnected world, connectivity’s reliable and widespread availability is increasingly important across various domains. This is especially more crucial for applications like the Internet of Things (IoT), remote sensing, disaster management, and bridging the digital divide. However, allocating the limited network resources efficiently and ensuring seamless handover between satellite and terrestrial networks present significant challenges. Therefore, this study introduces a resource allocation framework for integrated satellite–terrestrial networks to address these challenges. The framework leverages local cache pool deployments and non-orthogonal multiple access (NOMA) to reduce time delays and improve energy efficiency. Our proposed approach utilizes a multi-agent enabled deep deterministic policy gradient algorithm (MADDPG) to optimize user association, cache design, and transmission power control, resulting in enhanced energy efficiency. The approach comprises two phases: User Association and Power Control, where users are treated as agents, and Cache Optimization, where the satellite (Bs) is considered the agent. Through extensive simulations, we demonstrate that our approach surpasses conventional single-agent deep reinforcement learning algorithms in addressing cache design and resource allocation challenges in integrated terrestrial–satellite networks. Specifically, our proposed approach achieves significantly higher energy efficiency and reduced time delays compared to existing methods. This research highlights the importance and addresses the need for efficient resource allocation and cache design in integrated terrestrial–satellite networks, paving the way for enhanced connectivity and improved network performance in various applications.

Optimal wideband sequential sensing in cognitive radios via deep reinforcement learning

AbstractIn cognitive radios, wideband sequential sensing plays an important role, which is able to quickly identify temporary available transmission opportunities by adaptively allocating sensing resources. This paper proposes a Markov decision process for modelling the optimal control of sequential sensing, which provides a general formulation capturing various practical features, including sampling cost, sensing requirement, sensing budget etc. For solving the optimal sensing policy, a model‐augmented deep reinforcement learning algorithm is proposed, which enjoys high learning stability and efficiency, compared to conventional reinforcement learning algorithms.

Decision-making models on perceptual uncertainty with distributional reinforcement learning

Decision-making for autonomous vehicles in the presence of obstacle occlusions is difficult because the lack of accurate information affects the judgment. Existing methods may lead to overly conservative strategies and time-consuming computations that cannot be balanced with efficiency. We propose to use distributional reinforcement learning to hedge the risk of strategies, optimize the worse cases, and improve the efficiency of the algorithm so that the agent learns better actions. A batch of smaller values is used to replace the average value to optimize the worse case, and combined with frame stacking, we call it Efficient-Fully parameterized Quantile Function (E-FQF). This model is used to evaluate signal-free intersection crossing scenarios and makes more efficient moves and reduces the collision rate compared to conventional reinforcement learning algorithms in the presence of perceived occlusion. The model also has robustness in the case of data loss compared to the method with embedded long and short term memory.

A practical deep reinforcement learning framework for multivariate occupant-centric control in buildings

Reinforcement learning (RL) has been shown to have the potential for optimal control of heating, ventilation, and air conditioning (HVAC) systems. Although research on RL-based building control has received extensive attention in recent years, there is limited real-world implementation to evaluate its performance while keeping occupants in the loop. Additionally, many HVAC systems consist of multiple subsystems, but conventional RL algorithms face significant challenges when dealing with high-dimensional action spaces. This study proposes a practical deep reinforcement learning (DRL) based multivariate occupant-centric control framework that considers personalized thermal comfort and occupant presence. Specifically, Branching Dueling Q-network (BDQ) is leveraged as the learning agent to efficiently solve the multi-dimensional control task, and a tabular-based personal comfort modeling method is applied that is naturally integrated into human-in-the-loop operations. The BDQ agent is pre-trained in a virtual environment, followed by online deployment in a real office space for 5-dimensional action control. Based on the actual deployment and real-time comfort votes, our results showed a 14% reduction in cooling energy and an 11% improvement in total thermal acceptability.

Real-Time Battery Thermal Management for Electric Vehicles Based on Deep Reinforcement Learning

With the rapid developments of electric vehicles (EVs) in recent years, it is desirable to improve the energy efficiency to prolong the limited life of battery and extend the cruising range of EVs. In real EVs, the battery thermal management system is installed to cool the battery and maintain the expected high power output. In this article, we propose a novel energy management strategy based on deep reinforcement learning (DRL) considering battery thermal effects on energy efficiency. The main idea is to formulate energy management as an optimization problem, further extract the state sequence features of the vehicle via gated recurrent unit (GRU), and finally, propose a double deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> network (double DQN)-based algorithm to obtain the optimal strategy. Comparisons of our double DQN algorithm and existent fuzzy control, as well as two other conventional reinforcement learning (RL) algorithms, are conducted under New European Driving Cycle, FTP-75, HWFET, and US06 cycles, and the results demonstrate that the proposed algorithm achieves an energy reduction of more than 6.7% during aggressive driving.

Learning-Based Joint Optimization of Energy Delay and Privacy in Multiple-User Edge-Cloud Collaboration MEC Systems

The emergence of mobile edge computing (MEC) enables resource-limited user devices to run computation-intensive applications with the aid of edge server, but the untrustworthiness of the third-party edge server raises the leakage risk of users’ privacy. In this article, we consider an edge-cloud collaboration (ECC) scenario, consisting of multiple user devices with energy harvesting component, one edge server and one cloud server, and we concern about the issues of time latency, energy consumption, and privacy level of user devices in the process of task offloading. Specifically, we formulate the tradeoff between offloading cost and privacy level as a jointly optimization problem, and further model it as a Markov decision process (MDP). We then propose a privacy-preserving task offloading building upon the deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (DQN), which enables user devices to make the optimal offloading decision to decrease delay, reduce energy cost, and enhance privacy level. Extensive simulation results prove that our method can reduce the offloading cost whilst boosting the privacy level of user devices compared to conventional reinforcement learning (RL) algorithm and two baselines.

Bayesian Distributional Policy Gradients

Distributional Reinforcement Learning (RL) maintains the entire probability distribution of the reward-to-go, i.e. the return, providing more learning signals that account for the uncertainty associated with policy performance, which may be beneficial for trading off exploration and exploitation and policy learning in general. Previous works in distributional RL focused mainly on computing the state-action-return distributions, here we model the state-return distributions. This enables us to translate successful conventional RL algorithms that are based on state values into distributional RL. We formulate the distributional Bellman operation as an inference-based auto-encoding process that minimises Wasserstein metrics between target/model return distributions. The proposed algorithm, BDPG (Bayesian Distributional Policy Gradients), uses adversarial training in joint-contrastive learning to estimate a variational posterior from the returns. Moreover, we can now interpret the return prediction uncertainty as an information gain, which allows to obtain a new curiosity measure that helps BDPG steer exploration actively and efficiently. We demonstrate in a suite of Atari 2600 games and MuJoCo tasks, including well known hard-exploration challenges, how BDPG learns generally faster and with higher asymptotic performance than reference distributional RL algorithms.

Deep reinforcement learning framework for resilience enhancement of distribution systems under extreme weather events

More and more extreme weather events now occur due to global warming, and the existing power systems are powerless to cope with such high-impact, low-probability events. In order to tackle such highly stochastic events with large state spaces during an extreme event, conventional model-based approaches establish optimization models to capture the impact on the power system. However, the sophisticated models cause high computational complexity and do not have the ability to learn. This paper formulates the distributed generator (DG) rescheduling problem as a discrete Markov decision process (MDP) without transition probability information. The uncertainties associated with component failures are considered. A model-free optimization framework based on deep reinforcement learning (DRL) is proposed to determine the optimal rescheduling strategy to improve the resilience of a distribution system. Deep neural networks (DNN) are applied to extract large stochastic state space features and form multivariate Gaussian distributions to handle high-dimensional continuous control actions. A conventional reinforcement learning (RL) algorithm and two DRL algorithms are tested and compared in the IEEE 9-bus, IEEE 39-bus and IEEE 123-bus systems, and the results illustrate the effectiveness of the proposed framework in both meshed and radial topologies.

Agent-Based Modeling in Electricity Market Using Deep Deterministic Policy Gradient Algorithm

Game theoretic methods and simulations based on reinforcement learning (RL) are often used to analyze electricity market equilibrium. However, the former is limited to a simple market environment with complete information, and difficult to visually reflect the tacit collusion; while the conventional RL algorithm is limited to low-dimensional discrete state and action spaces, and the convergence is unstable. To address the aforementioned problems, this paper adopts deep deterministic policy gradient (DDPG) algorithm to model the bidding strategies of generation companies (GenCos). Simulation experiments, including different settings of GenCo, load and network, demonstrate that the proposed method is more accurate than conventional RL algorithm, and can converge to the Nash equilibrium of complete information even in the incomplete information environment. Moreover, the proposed method can intuitively reflect the different tacit collusion level by quantitatively adjusting GenCos' patience parameter, which can be an effective means to analyze market strategies.

Efficient Handover Mechanism for Radio Access Network Slicing by Exploiting Distributed Learning

Network slicing is identified as a fundamental architectural technology for future mobile networks since it can logically separate networks into multiple slices and provide tailored quality of service (QoS). However, the introduction of network slicing into radio access networks (RAN) can greatly increase user handover complexity in cellular networks. Specifically, both physical resource constraints on base stations (BSs) and logical connection constraints on network slices (NSs) should be considered when making a handover decision. Moreover, various service types call for an intelligent handover scheme to guarantee the diversified QoS requirements. As such, in this article, a multi-agent reinforcement LEarning based Smart handover Scheme, named LESS, is proposed, with the purpose of minimizing handover cost while maintaining user QoS. Due to the large action space introduced by multiple users and the data sparsity caused by user mobility, conventional reinforcement learning algorithms cannot be applied directly. To solve these difficulties, LESS exploits the unique characteristics of slicing in designing two algorithms: 1) LESS-DL, a distributed ${Q}$ -learning algorithm to make handover decisions with reduced action space but without compromising handover performance; 2) LESS-QVU, a modified ${Q}$ -value update algorithm which exploits slice traffic similarity to improve the accuracy of ${Q}$ -value evaluation with limited data. Thus, LESS uses LESS-DL to choose the target BS and NS when a handover occurs, while ${Q}$ -values are updated by using LESS-QVU. The convergence of LESS is theoretically proved in this article. Simulation results show that LESS can significantly improve network performance. In more detail, the number of handovers, handover cost and outage probability are reduced by around 50%, 65%, and 45%, respectively, when compared with traditional methods.

Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm

Balancing energy generation and consumption is essential for smoothing the power grids. The mismatch between energy supply and demand would not only increase the cost on both sides, but also has a great impact on the stability of the system. This paper proposes a novel energy sharing mechanism (ESM) to facilitate the consumption of local energy. With the help of the ESM, multiple prosumers have an opportunity to share surplus energy with neighboring prosumers. The problem is formulated as a leader–follower framework based on the Stackelberg game theory. To address the aforementioned problems, a deep deterministic policy gradient (DDPG) is applied to solve the Nash equilibrium (NE). The numerical results demonstrate that the proposed method is more stable than the conventional reinforcement learning (RL) algorithm. Moreover, the proposed method can converge to NE and find a relatively good energy sharing (ES) pricing strategy without knowing the specific system information. In short, it is notable that the proposed ESM can be seen as a win–win strategy for both prosumers and the power system.

A deep reinforcement learning method for managing wind farm uncertainties through energy storage system control and external reserve purchasing

In deregulated environment, the wind power producers (WPPs) will face the challenge of how to increase their revenues under uncertainties of wind generation and electricity price. This paper proposes a method based on deep reinforcement learning (DRL) to address this issue. A data-driven controller that directly maps the input observations, i.e., the forecasted wind generation and electricity price, to the control actions of the wind farm, i.e., the charge/discharge schedule of the relevant energy storage system (ESS) and the reserve purchase schedule, is trained according to the method. By the well-trained controller, the influence of the uncertainties of wind power and electricity price on the revenue can be automatically involved and an expected optimal decision can be obtained. Furthermore, a targeted DRL algorithm, i.e., the Rainbow algorithm, is implemented to improve the effectiveness of the controller. Especially, the algorithm can overcome the limitation of the conventional reinforcement learning algorithms that the input states must be discrete, and thus the validity of the control strategy can be significantly improved. Simulation results illustrate that the proposed method can effectively cope with the uncertainties and bring high revenues to the WPPs.

Adaptation to environmental change using reinforcement learning for robotic salamander

In the paper, a reinforcement learning technique is applied to produce a central pattern generation-based rhythmic motion control of a robotic salamander while moving toward a fixed target. Since its action spaces are continuous and there are various uncertainties in an environment that the robot moves, it is difficult for the robot to apply a conventional reinforcement learning algorithm. In order to overcome this issue, a deep deterministic policy gradient among the deep reinforcement learning algorithms is adopted. The robotic salamander and the environments where it moves are realized using the Gazebo dynamic simulator under the robot operating system environment. The algorithm is applied to the robotic simulation for the continuous motions in two different environments, i.e., from a firm ground to a mud. Through the simulation results, it is verified that the robotic salamander can smoothly move toward a desired target by adapting to the environmental change from the firm ground to the mud. The gradual improvement in the stability of learning algorithm is also confirmed through the simulations.

Deep Forest Reinforcement Learning for Preventive Strategy Considering Automatic Generation Control in Large-Scale Interconnected Power Systems

To reduce occurrences of emergency situations in large-scale interconnected power systems with large continuous disturbances, a preventive strategy for the automatic generation control (AGC) of power systems is proposed. To mitigate the curse of dimensionality that arises in conventional reinforcement learning algorithms, deep forest is applied to reinforcement learning. Therefore, deep forest reinforcement learning (DFRL) as a preventive strategy for AGC is proposed in this paper. The DFRL method consists of deep forest and multiple subsidiary reinforcement learning. The deep forest component of the DFRL is applied to predict the next systemic state of a power system, including emergency states and normal states. The multiple subsidiary reinforcement learning component, which includes reinforcement learning for emergency states and reinforcement learning for normal states, is applied to learn the features of the power system. The performance of the DFRL algorithm was compared to that of 10 other conventional AGC algorithms on a two-area load frequency control power system, a three-area power system, and the China Southern Power Grid. The DFRL method achieved the highest control performance. With this new method, both the occurrences of emergency situations and the curse of dimensionality can be simultaneously reduced.

Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing

Mobile edge computing (a.k.a. fog computing) has recently emerged to enable in-situ processing of delay-sensitive applications at the edge of mobile networks. Providing grid power supply in support of mobile edge computing, however, is costly and even infeasible (in certain rugged or under-developed areas), thus mandating on-site renewable energy as a major or even sole power supply in increasingly many scenarios. Nonetheless, the high intermittency and unpredictability of renewable energy make it very challenging to deliver a high quality of service to users in energy harvesting mobile edge computing systems. In this paper, we address the challenge of incorporating renewables into mobile edge computing and propose an efficient reinforcement learning-based resource management algorithm, which learns on-the-fly the optimal policy of dynamic workload offloading (to the centralized cloud) and edge server provisioning to minimize the long-term system cost (including both service delay and operational cost). Our online learning algorithm uses a decomposition of the (offline) value iteration and (online) reinforcement learning, thus achieving a significant improvement of learning rate and run-time performance when compared to standard reinforcement learning algorithms such as Q-learning. We prove the convergence of the proposed algorithm and analytically show that the learned policy has a simple monotone structure amenable to practical implementation. Our simulation results validate the efficacy of our algorithm, which significantly improves the edge computing performance compared to fixed or myopic optimization schemes and conventional reinforcement learning algorithms.

A kernel based true online Sarsa(λ) for continuous space control problems

Reinforcement learning is an efficient learning method for the control problem by interacting with the environment to get an optimal policy. However, it al so faces challenges such as low convergence accuracy and slow convergence. More over, conventional reinforcement learning algorithms could hardly solve continuous control problems. The kernel-based method can accelerate convergence speed and improve convergence accuracy; and the policy gradient method is a good way to deal with continuous space problems. We proposed a Sarsa(?) version of true online time difference algorithm, named True Online Sarsa(?)(TOSarsa(?)), on the basis of the clustering-based sample specification method and selective kernel-based value function. The TOSarsa(?) algorithm has a consistent result with both the forward view and the backward view which ensures to get an optimal policy in less time. Afterwards we also combined TOSarsa(?) with heuristic dynamic programming. The experiments showed our proposed algorithm worked well in dealing with continuous control problem.