Reinforcement Learning Solution Research Articles

An Adaptive Energy Orchestrator for Cyberphysical Systems Using Multiagent Reinforcement Learning

Reducing carbon emissions is a critical issue for the near future as climate change is an imminent reality. To reduce our carbon footprint, society must change its habits and behaviours to optimise energy consumption, and the current progress in embedded systems and artificial intelligence has the potential to make this easier. The smart building concept and intelligent energy management are key points to increase the use of renewable sources of energy as opposed to fossil fuels. In addition, cyber-physical systems (CPSs) provide an abstraction of the management of services that allows the integration of both virtual and physical systems in a seamless control architecture. In this paper, we propose to use multiagent reinforcement learning (MARL) to model the CPS services control plane in a smart house, with the purpose of minimising, by shifting or shutdown services, the use of non-renewable energy (fuel generator) by exploiting solar production and batteries. Furthermore, our proposal dynamically adapts its behaviour in real time according to current and historic energy production, thus being able to handle occasional changes in energy production due to meteorological phenomena or unexpected energy consumption. In order to evaluate our proposal, we have developed an open-source smart building energy simulator and deployed our use case. Finally, several simulations with different configurations are evaluated to verify the performance. The simulation results show that the reinforcement learning solution outperformed the priority-based and the heuristic-based solutions in both power consumption and adaptability in all configurations.

Dynamic collaborative truck-drone delivery with en-route synchronization and random requests

Coordinated truck and drone delivery is gaining popularity in logistics as it can greatly reduce operation costs. However, existing studies on related operations management problems typically ignore the following important features: (i) the random appearance of requests, which require operators to dynamically respond to the requests; and (ii) the decisions of optimal launch and retrieval locations for trucks and drones instead of fixed to customer locations, which can significantly impact the overall time costs. To tackle these challenges, this study investigates the dynamic collaborative truck-drone routing problem with randomly arriving requests and synchronization on routes. We model the problem as a Markov Decision Process (MDP) and solve the MDP via a reinforcement learning (RL) approach. The proposed RL approach determines: (i) whether each request should be serviced upon arrival, (ii) which truck or drone should be assigned for the request, and (iii) the optimal en-route take-off and landing positions for paired trucks and drones. We further employ a framework of decentralized learning and centralized dispatching in RL to increase performance. Numerical experiments are conducted to assess the proposed solution approach on instances generated based on both the Solomon dataset and real-world operational data of a logistics operator in Singapore over several benchmark algorithms under various battery endurance levels of drones and distinct transportation scenarios including node-based dynamic collaborative truck-drone routing problem, dynamic non-collaborative truck and drone routing problem, and dynamic vehicle routing problem. The results show that our RL solution outperforms the benchmark algorithm in total profit by an average of 28.03 %, and our en-route takeoff and landing scenario outperforms the benchmark scenarios in total profit by an average of 8.43 % in multi-day instances. Additionally, compared to the traditional node-based landing scenario, employing our en-route takeoff and landing strategy can save 0.9 h/(drone*day) of waiting time on average.

Path planning of manure-robot cleaners using grid-based reinforcement learning

The use of a robot cleaner for manure removal improves housing conditions for dairy cows in the face of labor shortages. However, current robot cleaners follow programmed fixed routes without considering the dynamic behaviors of cows. This cleaning approach is less efficient and leads to more cow-robot encounters or collisions, thus affecting animal welfare. To address these issues, this paper (1) developed heatmap models for cow locations and defecation behaviors; (2) proposed a dynamic path planning approach for the manure robot cleaner using Grid-based Reinforcement Learning; (3) incorporated cow location information and defecation behavior into the path planning process; (4) compared the performance of the proposed approach with two different cleaning methods: the current fixed programmed cleaning in practice and the ideal path produced by simulated annealing for traveling salesman problem. The simulations mimic the situation in a barn at Dairy Campus of Wageningen Livestock Research located in Leeuwarden (the Netherlands). Obviously, the best performance was achieved when the route was executed without cows present, resulting in no cow-robot collision. However, with cows present, the proposed dynamic path planning strategy achieved a 67.6% reduction in cow-robot encounters while maintaining 85.4% of the cleaning performance compared to the current programmed fixed routes. Compared to the ideal path produced by simulated annealing for traveling salesman problem, the proposed dynamic path planning approach achieved 5% better cleaning performance, at the cost of 25% more cow-robot encounters due to its longer working path. We conclude the proposed grid-based Reinforcement Learning solution for manure robots in barns cleaned most efficient with the least interference with cow traffic.

Ex-RL: Experience-based reinforcement learning

Reinforcement learning (RL) has achieved significant success across various tasks. However, generalizing RL for similar tasks remains a challenge. This study leverages expertise from related tasks to introduce a novel algorithm, Ex-RL, for executing transfer learning in tabular RL. The methodology concentrates on abstracting previous experiences into descriptive data and utilizing such data for similar tasks. The research focuses on classic RL solutions for balancing and anti-balancing, which improve the sample efficiency of the learning process. Studies indicate that weak learners, such as Q-learning, require fewer learning episodes, resulting in a 50% improvement and a higher success rate in the learning process. An online virtual lab was developed to facilitate the execution of the experiments. The code is available at Github.

Assessment of the broader applicability of a smart agent in peer-to-peer energy trading: A full factorial analysis of a multi-agent reinforcement learning solution

To realise the clean energy transition, peer-to-peer (P2P) renewable energy sharing markets have been proposed as one possible solution for achieving such a goal and are recognised as a potential path to achieving other goals such as affordable and reliable energy. Existing studies have shown that coordination at the micro level can be achieved by employing such P2P market structures. A pressing question concerns how to set the trade price such that the community coordinates in a way that maximises social welfare. A solution to this question based on multi-agent reinforcement learning (MARL) has been provided as a proof-of-concept in a single environment. However, various factors such as climate and community scale have been shown to affect the collective performance in such energy-sharing communities. In this work, to test the wider applicability of the proposed solution, a full factorial experiment based on the factors of climate, community scale, and price mechanism, is conducted to ascertain the response of the community w.r.t. the outputs: community self-sufficiency, total net-loss, and income equality. In short, we find that a community stands an odds of 2 to 1 in higher savings by adopting a smart agent.

Intelligent Agents for Data Exploration

Data Exploration is an incremental process that helps users express what they want through a conversation with the data. Reinforcement Learning (RL) is one of the most notable approaches to automate data exploration and several solutions have been proposed. We first summarize some RL solutions that were built for different applications. In this context, various data exploration operators are leveraged including traditional roll-up and drill-down operations and text-based operations. An RL agent is trained to generate the best policy according to a hand-crafted reward function. The benefit of training RL policies for specific data exploration tasks has been demonstrated more than once for exploring finding a needle in a haystack, for serendipitous galaxy exploration, for helping a customer land on a satisfactory product, for helping a conference chair build a program committee in a stepwise fashion, for summarizing large datasets, etc. With the advent of Large Language Models and their ability to reason sequentially, it has become legitimate to ask the question: would LLMs and AI planning outperform an RL policy in data exploration? More specifically, would LLMs help circumvent retraining for new tasks and striking a balance between specificity and generality? This led us to designing LLM-powered approaches that introduce a new way of thinking about data exploration.

Curriculum learning empowered reinforcement learning for graph-based portfolio management: Performance optimization and comprehensive analysis

Portfolio management (PM) is a popular financial process that concerns the occasional reallocation of a particular quantity of capital into a portfolio of assets, with the main aim of maximizing profitability conditioned to a certain level of risk. Given the inherent dynamicity of stock exchanges and development for long-term performance, reinforcement learning (RL) has become a dominating solution for solving the problem of portfolio management in an automated and efficient manner. Nevertheless, the present RL-based PM methods just take into account the variations in prices of portfolio assets and the implications of price variations, while overlooking the significant relationships among different assets in the market, which are extremely valuable for managerial decisions. To close this gap, this paper introduces a novel deep model that combines two subnetworks; one to learn a temporal representation of historical prices using a refined temporal learner, while the other learns the relationships between different stocks in the market using a relation graph learner (RGL). Then, the above learners are integrated into the curriculum RL scheme for formulating the PM as a curriculum Markov Decision Process, in which an adaptive curriculum policy is presented to enable the agent to adaptively minimize risk value and maximize cumulative return. Proof-of-concept experiments are performed on data from three public stock indices (namely S&P500, NYSE, and NASDAQ), and the results demonstrate the efficiency of the proposed framework in improving the portfolio management performance over the competing RL solutions.

Learning to route and schedule links in reconfigurable networks

This paper considers networks with a reconfigurable topology with so called 60 GHz dynamic links that can be activated or disabled over time. A fundamental problem is to jointly determine which 60 GHz dynamic links are active and the route chosen by source nodes over time. To this end, this paper outlines a hierarchical deep reinforcement learning solution that can be used to compute the optimal policy that determines for each time slot (i) active dynamic links, and (ii) the route used by each source–destination pair. The results show that the proposed approach results in a maximum average queue length that is 80% shorter than non-learning methods.

Markov game based on reinforcement learning solution against cyber–physical attacks in smart grid

The rapid expansion of information networks has amplified vulnerabilities in cyberspace, particularly within the power grid and communication layers of smart grid systems. Safeguarding smart grid security has become imperative for users concerned about both online and offline privacy. Detecting attacks in smart grids is crucial for ensuring the safety and continuity of the electrical network. This paper introduces a novel stealth attack-defense game that examines the impact of these invasions on the availability, confidentiality, and integrity of phasor measurement units. The proposed system considers both attack and defense perspectives. Attackers focus on key parameters to covertly infiltrate system layers while defending the smart grid system requires advanced knowledge of data flow and critical insights to establish robust protections. To address this cybersecurity challenge, we propose a game approach based on a partially observed Markov game with an improved Shapley Q-value and a multi-agent reinforcement learning framework, implemented in cooperative, communicative, and dynamic environments. The evaluation results demonstrate the robustness of the proposed approach against coordinated attacks in smart grid environments. Additionally, the results emphasize the significance of attack distribution as a crucial trace for identifying the attack source. Our approach shows promise for addressing various types of attacks against smart grid systems.

POMDP inference and robust solution via deep reinforcement learning: an application to railway optimal maintenance

Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the unavailability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), typically benefit from the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of Transformers and long short-term memory networks, which constitute model-free RL solutions and work directly on the observation space, with an approach termed the belief-input method, which works on the belief space by exploiting the learned POMDP model for belief inference. We apply these methods to the real-world problem of optimal maintenance planning for railway assets and compare the results with the current real-life policy. We show that the RL policy learned by the belief-input method is able to outperform the real-life policy by yielding significantly reduced life-cycle costs.

Demand response for residential building heating: Effective Monte Carlo Tree Search control based on physics-informed neural networks

To reduce global carbon emissions and limit climate change, controlling energy consumption in buildings is an important piece of the puzzle. Here, we specifically focus on using a demand response (DR) algorithm to limit the energy consumption of a residential building's heating system while respecting user's thermal comfort. In that domain, Reinforcement learning (RL) methods have been shown to be quite effective. One such RL method is Monte Carlo Tree Search (MCTS), which has achieved impressive success in playing board games (go, chess). A particular advantage of MCTS is that its decision tree structure naturally allows to integrate exogenous constraints (e.g., by trimming branches that violate them), while conventional RL solutions need more elaborate techniques (e.g., indirectly by adding penalties in the cost/reward function, or through a backup controller that corrects constraint-violating actions). The main aim of this paper is to study the adoption of MCTS for building control, since this (to the best of our knowledge) has remained largely unexplored. A specific property of MCTS is that it needs a simulator component that can predict subsequent system states, based on actions taken. A straightforward data-driven solution is to use black-box neural networks (NNs). We will however extend a Physics-informed Neural Network (PiNN) model to deliver multi-timestep predictions, and show the benefit it offers in terms of lower prediction errors (−32% MAE) as well as better MCTS performance (−4% energy cost, +7% thermal comfort) compared to a black-box NN. A second contribution will be to extend a vanilla MCTS version to adopt the ideas applied in AlphaZero (i.e., using learned prior and value functions and an action selection heuristic) to obtain lower computational costs while maintaining control performance.

Enhancing military medical evacuation dispatching with armed escort management: comparing model-based reinforcement learning approaches

The military medical evacuation (MEDEVAC) dispatching problem involves determining optimal policies for evacuating combat casualties to maximize patient survivability during military operations. This study explores a variation of the MEDEVAC dispatching problem, focusing on controlling armed escorts using a Markov decision process (MDP) model and model-based reinforcement learning (RL) approaches. A discounted, continuous-time MDP model over an infinite horizon is developed to maximize the expected total discounted reward of the system. Two model-based RL solution approaches are proposed: one utilizing semi-gradient descent Q-learning and another employing semi-gradient descent SARSA. A computational example, set in western and central Africa during contingency operations, assesses the performance of the RL-generated policies against the myopic policy, which military medical planners currently employ. Solution quality is derived from expected response time, a crucial determinant of life-saving potential in MEDEVAC operations. The research also explores sensitivity analysis and excursion scenarios to evaluate the RL-generated policies further. By explicitly controlling armed escort assets, dispatching authorities can better manage the location and allocation of these resources throughout combat operations. The findings of this study have the potential to inform military medical planning, operations, and tactics, ultimately leading to improved MEDEVAC system performance and higher patient survivability rates.

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

Extensive research has been carried out on reinforcement learning methods. The core idea of reinforcement learning is to learn methods by means of trial and error, and it has been successfully applied to robotics, autonomous driving, gaming, healthcare, resource management, and other fields. However, when building reinforcement learning solutions at the edge, not only are there the challenges of data-hungry and insufficient computational resources but also there is the difficulty of a single reinforcement learning method to meet the requirements of the model in terms of efficiency, generalization, robustness, and so on. These solutions rely on expert knowledge for the design of edge-side integrated reinforcement learning methods, and they lack high-level system architecture design to support their wider generalization and application. Therefore, in this paper, instead of surveying reinforcement learning systems, we survey the most commonly used options for each part of the architecture from the point of view of integrated application. We present the characteristics of traditional reinforcement learning in several aspects and design a corresponding integration framework based on them. In this process, we show a complete primer on the design of reinforcement learning architectures while also demonstrating the flexibility of the various parts of the architecture to be adapted to the characteristics of different edge tasks. Overall, reinforcement learning has become an important tool in intelligent decision making, but it still faces many challenges in the practical application in edge computing. The aim of this paper is to provide researchers and practitioners with a new, integrated perspective to better understand and apply reinforcement learning in edge decision-making tasks.

Leader-Follower Based Online Reinforcement Learning Algorithm in Problem with Hierarchy Decision Makers

This paper presents an online integral reinforcement learning (RL) solution for problems with hierarchy decision makers. Specifically, we reformulate this model as a leader-follower game in which control input and deterministic disturbance act as decision makers at different levels of hierarchy: the control input plays the role of the leader while the disturbance plays the role of the follower. The main contributions of this paper can be summarized as follows. First, we introduce online RL to deal with systems that have partially unknown information, meaning that accurate dynamic information is not required. Second, we solve the leader-follower coupled Hamilton-Jacobi (HJ) and Riccati equations approximately online using the derived algorithm. Third, we provide turning laws for cost functions and controllers that ensure closed-loop stability simultaneously.

UAV-assisted NOMA for Enhancing ISAC: A Deep Reinforcement Learning Solution

UAV-assisted NOMA for Enhancing ISAC: A Deep Reinforcement Learning Solution

Learn Zero-Constraint-Violation Safe Policy in Model-Free Constrained Reinforcement Learning.

We focus on learning the zero-constraint-violation safe policy in model-free reinforcement learning (RL). Existing model-free RL studies mostly use the posterior penalty to penalize dangerous actions, which means they must experience the danger to learn from the danger. Therefore, they cannot learn a zero-violation safe policy even after convergence. To handle this problem, we leverage the safety-oriented energy functions to learn zero-constraint-violation safe policies and propose the safe set actor-critic (SSAC) algorithm. The energy function is designed to increase rapidly for potentially dangerous actions, locating the safe set on the action space. Therefore, we can identify the dangerous actions prior to taking them and achieve zero-constraint violation. Our major contributions are twofold. First, we use the data-driven methods to learn the energy function, which releases the requirement of known dynamics. Second, we formulate a constrained RL problem to solve the zero-violation policies. We prove that our Lagrangian-based constrained RL solutions converge to the constrained optimal zero-violation policies theoretically. The proposed algorithm is evaluated on the complex simulation environments and a hardware-in-loop (HIL) experiment with a real autonomous vehicle controller. Experimental results suggest that the converged policies in all environments achieve zero-constraint violation and comparable performance with model-based baseline.

Knowledge transfer in multi-objective multi-agent reinforcement learning via generalized policy improvement

Even though many real-world problems are inherently distributed and multi-objective, most of the reinforcement learning (RL) literature deals with single agents and single objectives. While some of these problems can be solved using a single-agent single-objective RL solution (e.g., by specifying preferences over objectives), there are robustness issues, as well the fact that preferences may change over time, or it might not even be possible to set such preferences. Therefore, a need arises for a way to train multiple agents for any given preference distribution over the objectives. This work thus proposes a multi-objective multi-agent reinforcement learning (MOMARL) method in which agents build a shared set of policies during training, in a decentralized way, and then combine these policies using a generalization of policy improvement and policy evaluation (fundamental operations of RL algorithms) to generate effective behaviors for any possible preference distribution, without requiring any additional training. This method is applied to two different application scenarios: a multi-agent extension of a domain commonly used in the related literature, and traffic signal control, which is more complex, inherently distributed and multi-objective (the flow of both vehicles and pedestrians are considered). Results show that the approach is able to effectively and efficiently generate behaviors for the agents, given any preference over the objectives.

An efficient frame preemption algorithm for time-sensitive networks using enhanced graph convolutional network with particle swarm optimization

Numerous time-sensitive applications are being made possible by the Internet of Things (IoT) phenomenon, which calls for real-time transmission to provide communication services. Applications that depend on low latency have been made possible because of the 5G Ultra-Reliable and Low-Latency Communication (URLLC) scenario. Widely regarded as a possible paradigm for achieving the deterministic transmission guarantees for 5G, Time-Sensitive Networking (TSN) has gained much attention. TSN, on the other hand, is a hybrid traffic system that combines best-effort and time-sensitive traffic, necessitating efficient routing and frame preemption to provide a predictable and limited latency. On the other hand, time-sensitive and non-time-sensitive traffic optimization significantly broaden the issue area and increase the difficulty of establishing a solution. This research presents Enhanced Graph Convolutional Network-based Deep Reinforcement Learning (EGCN-based DRL) solutions for the joint optimization problem in real-world communication contexts. The EGCN is included in deep reinforcement learning to obtain the network's spatial dependency and enhance the generalization performance of the suggested technique (DRL). Specifically, the EGCN approximates the graph convolution kernel using the first-order Chebyshev polynomial, which reduces the complexity of the algorithm and improves the feasibility of the task. Particle Swarm Optimization (PSO) is also employed to hasten the convergence of the model training process. The objective of TSN is to provide deterministic and time-critical communication capabilities over Ethernet networks. TSN aims to enable reliable, low-latency, and synchronized communication between devices, ensuring that time-sensitive data and control messages are delivered with high precision and determinism. The suggested EGCN-based DRL algorithm surpasses cutting-edge techniques in terms of average end-to-end latency, and it also converges quickly, according to numerical simulations.

Optimal Computation Resource Allocation in Energy-Efficient Edge IoT Systems With Deep Reinforcement Learning

This paper investigates a computation resource optimization problem of mobile edge computing (MEC)-aided Internet-of-Things (IoT) devices with a reinforcement learning (RL) solution. Specifically, we leverage the stochastic optimization method and formulate the Lyapunov optimization technique to maximize the long-term energy efficiency, taking into account the transmission power, network stability, and transmission latency. Based on the Markov decision process and model-free deep RL (DRL) approach, we propose a double DRL-based online computation offloading method to implement a deep neural network that learns from interactions to solve the computation offloading and transmission latency problem in the dynamic MEC-aided IoT environments. Furthermore, we design an adaptive method for continuous action-state spaces to minimize the completion time and total energy consumption of the IoT devices for stochastic computation offloading tasks. The proposed real-time Lyapunov optimization and DRL algorithms achieve low computational complexity and optimal processing time. Simulation results demonstrate that the proposed algorithm can achieve near-optimal control performance with enhanced energy efficiency performance compared to the baseline policy control algorithms.

Heterogeneous GNN-RL-Based Task Offloading for UAV-Aided Smart Agriculture

Having unmanned aerial vehicles (UAVs) with edge computing capability hover over smart farmlands supports Internet of Things (IoT) devices with low processing capacity and power to accomplish their deadline-sensitive tasks efficiently and economically. In this work, we propose a graph neural network-based reinforcement learning solution to optimize the task offloading from these IoT devices to the UAVs. We conduct evaluations to show that our approach reduces task deadline violations while also increasing the mission time of the UAVs by optimizing their battery usage. Moreover, the proposed solution has increased robustness to network topology changes and is able to adapt to extreme cases, such as the failure of a UAV.