Multi-objective Reinforcement Learning Research Articles

Using hybrid multiobjective machine learning to optimise sonobuoy placement patterns

AbstractThis paper presents a new approach to finding optimal patterns for the placement of fields of sonobuoys in a complex undersea environment. We model the problem as a biobjective one, where the aim is to minimise both sensor placement time and uncertainty over target localisation. Both objectives may be important in time‐critical localisation scenarios and our approach allows an operator to choose between different optimal solutions, favouring lower placement time or lower localisation uncertainty as operational circumstances require. We develop a two‐phase algorithm, where an offline multiobjective evolutionary phase finds initial Pareto‐non‐dominated solutions to a static problem and then an online multiobjective reinforcement learning phase finds improved solutions using updated information. We find that the evolutionary algorithm improves significantly on standard grid patterns and that the reinforcement learning algorithm improves further on the evolutionary phase. The number of sonobuoys required may also be reduced.

Multi-objective reinforcement learning framework for dynamic flexible job shop scheduling problem with uncertain events

The economic benefits for manufacturing companies will be influenced by how it handles potential dynamic events and performs multi-objective real-time scheduling for existing dynamic events. Based on these, we propose a new dynamic multi-objective flexible job shop scheduling problem (DMFJSP) to simulate realistic production environment. Six dynamic events are involved in the problem including job insertion, job cancellation, job operation modification, machine addition, machine tool replacement and machine breakdown. As well as three objectives of longest job processing time (makespan), average machine utilization and average job processing delay rate with a set of constraints are also raised in the study. Then this research designs a novel dynamic multi-objective scheduling algorithm based on deep reinforcement learning. The algorithm uses two deep Q-learning networks and a real-time processing framework to process each dynamic event and generate complete scheduling scheme. In addition, an improved local search algorithm is adopted to further optimize the scheduling results and the idea of combination is used to make the scheduling rules more comprehensive. Experiments on 27 instances show the superiority and stability of our approach compared to each proposed combined rule, well-known scheduling rules and standard deep Q-learning based algorithms. Compared to the current optimal deep Q-learning method, the maximum performance improvement for our three objectives are approximately 57%, 164% and 28%.

Multi-objective fuzzy Q-learning to solve continuous state-action problems

Many real world problems are multi-objective. Thus, the need for multi-objective learning and optimization algorithms is inevitable. Although the multi-objective optimization algorithms are well-studied, the multi-objective learning algorithms have attracted less attention. In this paper, a fuzzy multi-objective reinforcement learning algorithm is proposed, and we refer to it as the multi-objective fuzzy Q-learning (MOFQL) algorithm. The algorithm is implemented to solve a bi-objective reach-avoid game. The majority of the multi-objective reinforcement algorithms proposed address solving problems in the discrete state-action domain. However, the MOFQL algorithm can also handle problems in a continuous state-action domain. A fuzzy inference system (FIS) is implemented to estimate the value function for the bi-objective problem. We used a temporal difference (TD) approach to update the fuzzy rules. The proposed method is a multi-policy multi-objective algorithm and can find the non-convex regions of the Pareto front.

Multi-Objective Resource Scheduling for IoT Systems Using Reinforcement Learning

IoT embedded systems have multiple objectives that need to be maximized simultaneously. These objectives conflict with each other due to limited resources and tradeoffs that need to be made. This requires multi-objective optimization (MOO) and multiple Pareto-optimal solutions are possible. In such a case, tradeoffs are made w.r.t. a user-defined preference. This work presents a general Multi-objective Reinforcement Learning (MORL) framework for MOO of IoT embedded systems. This framework comprises a general Multi-objective Markov Decision Process (MOMDP) formulation and two novel low-compute MORL algorithms. The algorithms learn policies to tradeoff between multiple objectives using a single preference parameter. We take the energy scheduling problem in general Energy Harvesting Wireless Sensor Nodes (EHWSNs) as a case example in which a sensor node is required to maximize its sensing rate, and transmission performance as well as ensure long-term uninterrupted operation within a very tight energy budget. We simulate single-task and dual-task EHWSN systems to evaluate our framework. The results demonstrate that our MORL algorithms can learn better policies at lower learning costs and successfully tradeoff between multiple objectives at runtime.

Learning to Optimise a Swarm of UAVs

The use of Unmanned Aerial Vehicles (UAVs) has shown a drastic increase in interest in the past few years. Current applications mainly depend on single UAV operations, which face critical limitations such as mission range or resilience. Using several autonomous UAVs as a swarm is a promising approach to overcome these. However, designing an efficient swarm is a challenging task, since its global behaviour emerges solely from local decisions and interactions. These properties make classical multirobot design techniques not applicable, while evolutionary swarm robotics is typically limited to a single use case. This work, thus, proposes an automated swarming algorithm design approach, and more precisely, a generative hyper-heuristic relying on multi-objective reinforcement learning, that permits us to obtain not only efficient but also reusable swarming behaviours. Experimental results on a three-objective variant of the Coverage of a Connected UAV Swarm problem demonstrate that it not only permits one to generate swarming heuristics that outperform the state-of-the-art in terms of coverage by a swarm of UAVs but also provides high stability. Indeed, it is empirically demonstrated that the model trained on a certain class of instances generates heuristics and is capable of performing well on instances with a different size or swarm density.

Multiobjective Load Balancing for Multiband Downlink Cellular Networks: A Meta- Reinforcement Learning Approach

Load balancing has become a key technique to handle the increasing traffic demand and improve the user experience. It evenly distributes the traffic across network resources by offloading users from overloaded base stations or channels to less crowded ones. Load balancing is a multi-objective optimization problem involving the automatic adjustment of several parameters to simultaneously maximize multiple network performance indicators. However, the existing methods mostly rely on single-objective approaches which lead to sub-optimal solutions. In this paper, we introduce the first multi-objective reinforcement learning (MORL) framework for load balancing. Specifically, we propose a solution based on meta-reinforcement learning (meta-RL) to learn a general policy capable of quickly adapting to new trade-offs between the objectives. We further enhance the generalization of our proposed solution using policy distillation techniques. To showcase the effectiveness of our framework, experiments are conducted based on real-world traffic scenarios. Our results show that our load balancing framework can (i) significantly outperform the existing rule-based and single-objective solutions, (ii) compute better Pareto front approximations compared to MORL baselines, and (iii) quickly adapt to new objective trade-offs.

An Improved Multi-Objective Deep Reinforcement Learning Algorithm Based on Envelope Update

Multi-objective reinforcement learning (MORL) aims to uniformly approximate the Pareto frontier in multi-objective decision-making problems, which suffers from insufficient exploration and unstable convergence. We propose a multi-objective deep reinforcement learning algorithm (envelope with dueling structure, Noisynet, and soft update (EDNs)) to improve the ability of the agent to learn optimal multi-objective strategies. Firstly, the EDNs algorithm uses neural networks to approximate the value function and update the parameters based on the convex envelope of the solution boundary. Then, the DQN structure is replaced with the dueling structure, and the state value function is split into the dominance function and value function to make it converge faster. Secondly, the Noisynet method is used to add exploration noise to the neural network parameters to make the agent have a more efficient exploration ability. Finally, the soft update method updates the target network parameters to stabilize the training procedure. We use the DST environment as a case study, and the experimental results show that the EDNs algorithm has better stability and exploration capability than the EMODRL algorithm. In 1000 episodes, the EDNs algorithm improved the coverage by 5.39% and reduced the adaptation error by 36.87%.

Joint Optimization of Concave Scalarized Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithm

Many engineering problems have multiple objectives, and the overall aim is to optimize a non-linear function of these objectives. In this paper, we formulate the problem of maximizing a non-linear concave function of multiple long-term objectives. A policy-gradient based model-free algorithm is proposed for the problem. To compute an estimate of the gradient, an asymptotically biased estimator is proposed. The proposed algorithm is shown to achieve convergence to within an ε of the global optima after sampling O(M4 σ2/(1-γ)8ε4) trajectories where γ is the discount factor and M is the number of the agents, thus achieving the same dependence on ε as the policy gradient algorithm for the standard reinforcement learning.

A multi-objective reinforcement learning approach for resequencing scheduling problems in automotive manufacturing systems

This study investigated a multi-objective resequencing scheduling problem in the automotive manufacturing systems due to operational requirements on the color-batching of the paint shop and sequential requirements on the sequence adherence of an assembly shop. Resequencing cars as color-oriented batches reduced the costs of color changes and operational costs for paint shops. Also, assembly shops required paint shops to complete cars with fewer delays so that high sequence adherence with its demand was assured. Based on real-world applications, we investigated two contradictory objectives-color change costs and sequence tardiness-in a single-machine flowshop scheduling environment. A multi-objective-deep-Q-network algorithm was developed to determine the Pareto frontier. Reward shaping was designed to improve the convergence of the neural network. The 2D-folded-normal distribution was designed to sample the preference, which made the exploration and exploitation of the neural network more comprehensive and improved the training efficiency. Two experiments were conducted and showed that the proposed approach outperformed the meta-heuristic algorithm and the envelope Q-learning algorithm in solving time, performance, the convergence of the neural network, and the diversity of the Pareto frontier. Therefore, the proposed approach can be used in automotive paint shops to improve scheduling efficiency and reduce operational costs.

Expected scalarised returns dominance: a new solution concept for multi-objective decision making

AbstractIn many real-world scenarios, the utility of a user is derived from a single execution of a policy. In this case, to apply multi-objective reinforcement learning, the expected utility of the returns must be optimised. Various scenarios exist where a user’s preferences over objectives (also known as the utility function) are unknown or difficult to specify. In such scenarios, a set of optimal policies must be learned. However, settings where the expected utility must be maximised have been largely overlooked by the multi-objective reinforcement learning community and, as a consequence, a set of optimal solutions has yet to be defined. In this work, we propose first-order stochastic dominance as a criterion to build solution sets to maximise expected utility. We also define a new dominance criterion, known as expected scalarised returns (ESR) dominance, that extends first-order stochastic dominance to allow a set of optimal policies to be learned in practice. Additionally, we define a new solution concept called the ESR set, which is a set of policies that are ESR dominant. Finally, we present a new multi-objective tabular distributional reinforcement learning (MOTDRL) algorithm to learn the ESR set in multi-objective multi-armed bandit settings.

A Multi-Objective Reinforcement Learning Based Controller for Autonomous Navigation in Challenging Environments

In this paper, we introduce a self-trained controller for autonomous navigation in static and dynamic (with moving walls and nets) challenging environments (including trees, nets, windows, and pipe) using deep reinforcement learning, simultaneously trained using multiple rewards. We train our RL algorithm in a multi-objective way. Our algorithm learns to generate continuous action for controlling the UAV. Our algorithm aims to generate waypoints for the UAV in such a way as to reach a goal area (shown by an RGB image) while avoiding static and dynamic obstacles. In this text, we use the RGB-D image as the input for the algorithm, and it learns to control the UAV in 3-DoF (x, y, and z). We train our robot in environments simulated by Gazebo sim. For communication between our algorithm and the simulated environments, we use the robot operating system. Finally, we visualize the trajectories generated by our trained algorithms using several methods and illustrate our results that clearly show our algorithm’s capability in learning to maximize the defined multi-objective reward.

Multi-objective reinforcement learning for fed-batch fermentation process control

Many real-world control problems involve conflicting objectives. For different objectives, it is necessary to obtain Pareto optimal solution sets for each one. Over recent years, multi-objective reinforcement learning (MORL) has been extensively studied to solve this problem. However, the multi-objective optimization problem of complex continuous control processes still requires exploration. Soft proximal policy optimization algorithms have also been proposed to be combined with a hybrid weight-generation method for application to find the Pareto front approximation of the fed-batch fermentation process. This algorithm intends to initially find a single policy for the multi-objective reinforcement learning problem. A hybrid weight-generation method is then used to change the weights between different objectives to find a set of Pareto optimal solutions. In addition, we analyzed the mechanism of the fed-batch process, established the kinetic model, and designed the experimental environment based on the OpenAI Gym library. Experimental results showed that the proposed algorithm is effective and efficient in approaching the Pareto front of the fed-batch fermentation problem.

Multiobjective Reinforcement Learning Based Energy Consumption in C-RAN enabled Massive MIMO

Multiobjective Reinforcement Learning Based Energy Consumption in C-RAN enabled Massive MIMO

A practical guide to multi-objective reinforcement learning and planning

Real-world sequential decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems.

Nondominated Policy-Guided Learning in Multi-Objective Reinforcement Learning

Control intelligence is a typical field where there is a trade-off between target objectives, and researchers in this field have longed for artificial intelligence that achieves the target objectives. Multi-objective deep reinforcement learning was sufficient to satisfy this need. In particular, multi-objective deep reinforcement learning methods based on policy optimization are leading the optimization of control intelligence. However, multi-objective reinforcement learning has difficulties when finding various Pareto optimals of multi-objectives due to the greedy nature of reinforcement learning. We propose a method of policy assimilation to solve this problem. This method was applied to MO-V-MPO, one of preference-based multi-objective reinforcement learning, to increase diversity. The performance of this method has been verified through experiments in a continuous control environment.

Preference based multi-objective reinforcement learning for multi-microgrid system optimization problem in smart grid

Grid-connected microgrids comprising renewable energy, energy storage systems and local load, play a vital role in decreasing the energy consumption of fossil diesel and greenhouse gas emissions. A distribution power network connecting several microgrids can promote more potent and reliable operations to enhance the security and privacy of the power system. However, the operation control for a multi-microgrid system is a big challenge. To design a multi-microgrid power system, an intelligent multi-microgrids energy management method is proposed based on the preference-based multi-objective reinforcement learning (PMORL) techniques. The power system model can be divided into three layers: the consumer layer, the independent system operator layer, and the power grid layer. Each layer intends to maximize its benefit. The PMORL is proposed to lead to a Pareto optimal set for each object to achieve these objectives. A non-dominated solution is decided to execute a balanced plan not to favor any particular participant. The preference-based results show that the proposed method can effectively learn different preferences. The simulation outcomes confirm the performance of the PMORL and verify the viability of the proposed method.

Instilling moral value alignment by means of multi-objective reinforcement learning

AI research is being challenged with ensuring that autonomous agents learn to behave ethically, namely in alignment with moral values. Here, we propose a novel way of tackling the value alignment problem as a two-step process. The first step consists on formalising moral values and value aligned behaviour based on philosophical foundations. Our formalisation is compatible with the framework of (Multi-Objective) Reinforcement Learning, to ease the handling of an agent’s individual and ethical objectives. The second step consists in designing an environment wherein an agent learns to behave ethically while pursuing its individual objective. We leverage on our theoretical results to introduce an algorithm that automates our two-step approach. In the cases where value-aligned behaviour is possible, our algorithm produces a learning environment for the agent wherein it will learn a value-aligned behaviour.

Energy-Efficient Gait Optimization of Snake-Like Modular Robots by Using Multiobjective Reinforcement Learning and a Fuzzy Inference System

Snake-like modular robots (MRs) are highly flexible, but, to traverse a challenging terrain or explore a region of interest, MR needs to attain efficient locomotion depending on a tradeoff between objectives like forward velocity and power consumption of the robot. The objectives can vary with different weights depending upon the situation, reflecting relative objective importance. This study developed a multiobjective reinforcement learning algorithm based on a fuzzy inference system (FI-MORL) to select the most appropriate gait parameters of snake-like MRs according to the objective weights. The developed algorithm employs a fuzzy inference system to reduce the number of states in an environment, which results in faster learning. The proposed approach uses the previously learned experience to rapidly achieve the best objective values in response to a change in weights. While setting equal importance to the objectives, FI-MORL delivers superior performance than single-objective reinforcement learning algorithms by consuming 2% less power and gaining 2.5% higher velocity since it mitigates the effect of weight change, similar performance found comparing Actor-Critic algorithm. Likewise, the proposed method outperforms by consuming 14% less power and achieving 11% higher velocity than traditional methods like Proximal Policy Optimization, Deep Q-Network, and Vanilla Policy Gradient. Even after weight change, FI-MORL achieved a 14% higher reward than the above methods. The proposed FI-MORL framework can effectively converge quicker and efficiently handle the changes in objective weights.

An online scalarization multi-objective reinforcement learning algorithm: TOPSIS Q-learning

Abstract Conventional reinforcement learning focuses on problems with single objective. However, many problems have multiple objectives or criteria that may be independent, related, or contradictory. In such cases, multi-objective reinforcement learning is used to propose a compromise among the solutions to balance the objectives. TOPSIS is a multi-criteria decision method that selects the alternative with minimum distance from the positive ideal solution and the maximum distance from the negative ideal solution, so it can be used effectively in the decision-making process to select the next action. In this research a single-policy algorithm called TOPSIS Q-Learning is provided with focus on its performance in online mode. Unlike all single-policy methods, in the first version of the algorithm, there is no need for the user to specify the weights of the objectives. The user’s preferences may not be completely definite, so all weight preferences are combined together as decision criteria and a solution is generated by considering all these preferences at once and user can model the uncertainty and weight changes of objectives around their specified preferences of objectives. If the user only wants to apply the algorithm for a specific set of weights the second version of the algorithm efficiently accomplishes that.

Reinforcement Learning for Joint Optimization of Communication and Computation in Vehicular Networks

Ultra reliability and low latency communications (URLLC) are considered as one of the most important use cases for computation tasks in the Internet of Vehicles (IoV) edge computing networks. To meet the task requirements in IoV networks, communication and computing resources are allocated in an effective and efficient way. Thus, this paper proposes a multi-objective reinforcement learning strategy, called intelligent communication and computation resource allocation (ICCRA), which combines communication and computing resource allocation to reduce the total system cost consisting of latency and reliability. Specifically, this strategy can be decomposed into three algorithms. The algorithm called joint computation offloading and collaboration is a general framework of the strategy, it first uses the K-nearest neighbor method to select offloading layers for computation tasks, such as cloud computing layer, mobile edge computing layer, and local computing layer. Then, when selecting the local computing layer to perform the task, the algorithm called collaborative vehicle selection is used to find the target vehicle to execute collaborative computing. The allocation of communication and computing resources is defined as two independent objectives, the algorithm called multi-objective resource allocation uses the reinforcement learning to achieve the optimal solution, at the mobile edge computing layer. Simulation results show that compared with local computing, edge computing, and random computing strategies, the proposed strategy reduces the total system cost effectively. For instance, compared with edge computing strategy, the total system cost of the proposed strategy can be reduced about 50% on average with the different number of vehicle tasks.