Multi-objective Markov Decision Process Research Articles

Multi-objective task offloading for highly dynamic heterogeneous Vehicular Edge Computing: An efficient reinforcement learning approach

Vehicular Edge Computing (VEC) provides a flexible distributed computing paradigm for offloading computations to the vehicular network, which can effectively solve the problem of limited vehicle computing resources and meet the on-vehicle computing requests of users. However, the conflict of interest between vehicle users and service providers leads to the need to consider a variety of conflict optimization goals for computing offloading, and the dynamic nature of vehicle networks, such as vehicle mobility and time-varying network conditions, make the offloading effectiveness of vehicle computing requests and the adaptability to complex VEC scenarios challenging. To address these challenges, this paper proposes a multi-objective optimization model suitable for computational offloading of dynamic heterogeneous VEC networks. By formulating the dynamic multi-objective computational offloading problem as a multi-objective Markov Decision Process (MOMDP), this paper designs a novel multi-objective reinforcement learning algorithm EMOTO, which aims to minimize the average task execution delay and average vehicle energy consumption, and maximize the revenue of service providers. In this paper, a preference priority sampling module is proposed, and a model-augmented environment estimator is introduced to learn the environmental model for multi-objective optimization, so as to solve the problem that the agent is difficult to learn steadily caused by the highly dynamic change of VEC environment, thus to effectively realize the joint optimization of multiple objectives and improve the decision-making accuracy and efficiency of the algorithm. Experiments show that EMOTO has superior performance on multiple optimization objectives compared with advanced multi-objective reinforcement learning algorithms. In addition, the algorithm shows robustness when applied to different environmental settings and better adapting to highly dynamic environments, and balancing the conflict of interest between vehicle users and service providers.

Exploring the Pareto front of multi-objective COVID-19 mitigation policies using reinforcement learning

Infectious disease outbreaks can have a disruptive impact on public health and societal processes. As decision-making in the context of epidemic mitigation is multi-dimensional hence complex, reinforcement learning in combination with complex epidemic models provides a methodology to design refined prevention strategies. Current research focuses on optimizing policies with respect to a single objective, such as the pathogen’s attack rate. However, as the mitigation of epidemics involves distinct, and possibly conflicting, criteria (i.a., mortality, morbidity, economic cost, well-being), a multi-objective decision approach is warranted to obtain balanced policies. To enhance future decision-making, we propose a deep multi-objective reinforcement learning approach by building upon a state-of-the-art algorithm called Pareto Conditioned Networks (PCN) to obtain a set of solutions for distinct outcomes of the decision problem. We consider different deconfinement strategies after the first Belgian lockdown within the COVID-19 pandemic and aim to minimize both COVID-19 cases (i.e., infections and hospitalizations) and the societal burden induced by the mitigation measures. As such, we connected a multi-objective Markov decision process with a stochastic compartment model designed to approximate the Belgian COVID-19 waves and explore reactive strategies. As these social mitigation measures are implemented in a continuous action space that modulates the contact matrix of the age-structured epidemic model, we extend PCN to this setting. We evaluate the solution set that PCN returns, and observe that it explored the whole range of possible social restrictions, leading to high-quality trade-offs, as it captured the problem dynamics. In this work, we demonstrate that multi-objective reinforcement learning adds value to epidemiological modeling and provides essential insights to balance mitigation policies.

Robust Multiobjective Reinforcement Learning Considering Environmental Uncertainties.

Numerous real-world decision or control problems involve multiple conflicting objectives whose relative importance (preference) is required to be weighed in different scenarios. While Pareto optimality is desired, environmental uncertainties (e.g., environmental changes or observational noises) may mislead the agent into performing suboptimal policies. In this article, we present a novel multiobjective optimization paradigm, robust multiobjective reinforcement learning (RMORL) considering environmental uncertainties, to train a single model that can approximate robust Pareto-optimal policies across the entire preference space. To enhance policy robustness against environmental changes, an environmental disturbance is modeled as an adversarial agent across the entire preference space via incorporating a zero-sum game into a multiobjective Markov decision process (MOMDP). Additionally, we devise an adversarial defense technique against observational perturbations, which ensures that policy variations, perturbed by adversarial attacks on state observations, remain within bounds under any specified preferences. The proposed technique is assessed in five multiobjective environments with continuous action spaces, showcasing its effectiveness through comparisons with competitive baselines, which encompass classical and state-of-the-art schemes.

Personalized robotic control via constrained multi-objective reinforcement learning

Reinforcement learning is capable of providing state-of-art performance in end-to-end robotic control tasks. Nevertheless, many real-world control tasks necessitate the balancing of multiple conflicting objectives while simultaneously ensuring that the learned policies adhere to constraints. Additionally, individual users may typically prefer to explore the personalized and diversified robotic control modes via specific preferences. Therefore, this paper presents a novel constrained multi-objective reinforcement learning algorithm for personalized end-to-end robotic control with continuous actions, allowing a trained single model to approximate the Pareto optimal policies for any user-specified preferences. The proposed approach is formulated as a constrained multi-objective Markov decision process, incorporating a nonlinear constraint design to facilitate the agent in learning optimal policies that align with specified user preferences across the entire preference space. Meanwhile, a comprehensive index based on hypervolume and entropy is presented to measure the convergence, diversity and evenness of the learned control policies. The proposed scheme is evaluated on nine multi-objective end-to-end robotic control tasks with continuous action space, and its effectiveness is demonstrated in comparison with the competitive baselines, including classical and state-of-the-art algorithms.

Matching-Aided-Learning Resource Allocation for Dynamic Offloading in mmWave MEC System

With exploiting massive spectrum resources, millimeter wave (mmWave) communications significantly improve the offloading capability for future mobile edge computing (MEC) techniques, which however is constrained by blockage problem in dynamic environments. In this paper, we study the resource allocation problem for the conceived mmWave MEC system with dynamic offloading process, in which the UEs are characterized by being mobile and having the imperfect knowledge of the offloading tasks coming. By introducing the multi-objective Markov decision process (MOMDP), the resource allocation problem is modeled by simultaneously minimizing the delay and energy consumption, where jointly considering the multi-beam assignment (mBA) and beamwidth and power optimization (BPO). To tackle this problem, we innovatively propose a matching-aided-learning (MaL) resource allocation scheme, with the aid of a learnable weight based attention mechanism (LW-AM) for adapting the dynamic offloading process. In particular, our MaL scheme includes many-to-one matching (M2O-M) based mBA algorithm and deep deterministic policy gradient (DDPG) based BPO algorithm, which are executed iteratively and converge with relatively low number of iterations. The simulation results show the practical value of the proposed MaL, which can approach the performance of benchmark scheme with perfect knowledge of offloading tasks.

A multi-objective reinforcement learning approach for furniture arrangement with optimal IEQ in multi-occupant offices

Indoor Environmental Quality (IEQ) concerns several aspects of environmental comforts, such as thermal, visual and acoustics comfort. In particular, IEQ plays a relevant role in workers’ satisfaction since it strongly influences health, well-being, and productivity. Specifically, it has been demonstrated that the furniture configuration in working spaces affects the occupant’s comfort perception. Nevertheless, IEQ has been either neglected or partially addressed in the context of interior design. The contribution of this paper is to introduce a novel method for furniture layout optimisation in terms of IEQ requirements in multi-occupant offices. In particular, we explore the furniture arrangement task as a Multi-Objective Markov Decision Process (MOMDP), which is solved by a reinforcement learning (RL) agent. The goal is to determine optimal workstation positions that maximise workers’ IEQ satisfaction and functional aspects of working spaces under analysis. Firstly, we formulated the furniture layout task as a MOMDP problem by defining reward functions in terms of thermal, acoustics and visual comfort. Then, we train the RL agent to produce optimal/suboptimal layout patterns through a Q-learning-based algorithm. We conducted experiments in two different offices. The experimental results demonstrated that the proposed multi-objective RL approach is able to determine optimal furniture arrangements that provide a balance among office occupants in terms of IEQ satisfaction. Moreover, numerical results show that the proposed approach can be a valuable tool for evaluating the conformity to the environmental comfort standard of working environments during the furniture layout design phase instead of applying corrections during the post-occupancy evaluation.

Traffic efficiency and fairness optimisation for autonomous intersection management based on reinforcement learning

Autonomous Intersection Management (AIM) for high-level Connected and Automated Vehicles (CAVs) has evolved from rule-based to optimisation-based policies. However, at congested major-minor intersections, optimising solely for efficiency can negatively impact vehicle fairness. This study addresses this issue by proposing a deep reinforcement learning approach that optimises both traffic efficiency and fairness for AIM. In the modelled multi-objective Markov decision process, traffic fairness is measured by the difference between the crossing order and the approaching order of CAVs, while traffic efficiency is measured by average travel time. With unknown preferences of the objectives, Bellman optimality equation is generalised to obtain the optimal policies over the space of all possible preferences during the iterative training process. The effectiveness of the proposed method is evaluated in a simulated real-world intersection and compared with three benchmark policies, including the fairest policy for AIM: first-come-first-served. The learned policies perform best in reducing overall average vehicle delay, and demonstrate outstanding performance in balancing traffic fairness and efficiency.

Multi-objective deep reinforcement learning for computation offloading in UAV-assisted multi-access edge computing

Unmanned aerial vehicle-assisted multi-access edge computing (UAV-MEC) plays an important role in some complex environments such as mountainous and disaster areas. Computation offloading problem (COP) is one of the key issues of UAV-MEC, which mainly aims to minimize the conflict goals between energy consumption and delay. Due to the time-varying and uncertain nature of the UAV-MEC system, deep reinforcement learning is an effective method for solving the COP. Different from the existing works, in this paper, the COP in UAV-MEC system is modeled as a multi-objective Markov decision process, and a multi-objective deep reinforcement learning method is proposed to solve it. In the proposed algorithm, the scalar reward of reinforcement learning is expanded into a vector reward, and the weights are dynamically adjusted to meet different user preferences. The most important preferences are selected by non-dominated sorting, which can better maintain the previously learned strategy. In addition, the Q network structure combines Double Deep Q Network (Double DQN) with Dueling Deep Q Network (Dueling DQN) to improve the optimization efficiency. Simulation results show that the algorithm achieves a good balance between energy consumption and delay, and can obtain a better computation offloading scheme.

Same-day delivery with fair customer service

The demand for same-day delivery (SDD) has increased rapidly in the last few years and has particularly boomed during the COVID-19 pandemic. The fast growth is not without its challenge. In 2016, due to low concentrations of memberships and far distance from the depot, certain minority neighborhoods were excluded from receiving Amazon’s SDD service, raising concerns about fairness. In this paper, we study the problem of offering fair SDD service to customers. The service area is partitioned into different regions. Over the course of a day, customers request for SDD service, and the timing of requests and delivery locations are not known in advance. The dispatcher dynamically assigns vehicles to make deliveries to accepted customers before their delivery deadline. In addition to overall service rate (utility), we maximize the minimal regional service rate across all regions (fairness). We model the problem as a multi-objective Markov decision process and develop a deep Q-learning solution approach. We introduce a novel transformation of learning from rates to actual services, which creates a stable and efficient learning process. Computational results demonstrate the effectiveness of our approach in alleviating unfairness both spatially and temporally in different customer geographies. We show this effectiveness is valid with different depot locations, providing businesses with an opportunity to achieve better fairness from any location. We also show that the proposed approach performs efficiently when serving heterogeneously wealthy districts in the city.

Multi-Objective Resource Scheduling for IoT Systems Using Reinforcement Learning

IoT embedded systems have multiple objectives that need to be maximized simultaneously. These objectives conflict with each other due to limited resources and tradeoffs that need to be made. This requires multi-objective optimization (MOO) and multiple Pareto-optimal solutions are possible. In such a case, tradeoffs are made w.r.t. a user-defined preference. This work presents a general Multi-objective Reinforcement Learning (MORL) framework for MOO of IoT embedded systems. This framework comprises a general Multi-objective Markov Decision Process (MOMDP) formulation and two novel low-compute MORL algorithms. The algorithms learn policies to tradeoff between multiple objectives using a single preference parameter. We take the energy scheduling problem in general Energy Harvesting Wireless Sensor Nodes (EHWSNs) as a case example in which a sensor node is required to maximize its sensing rate, and transmission performance as well as ensure long-term uninterrupted operation within a very tight energy budget. We simulate single-task and dual-task EHWSN systems to evaluate our framework. The results demonstrate that our MORL algorithms can learn better policies at lower learning costs and successfully tradeoff between multiple objectives at runtime.

Neuro-Evolutionary Direct Policy Search for Multiobjective Optimal Control

Direct policy search (DPS) is emerging as one of the most effective and widely applied reinforcement learning (RL) methods to design optimal control policies for multiobjective Markov decision processes (MOMDPs). Traditionally, DPS defines the control policy within a preselected functional class and searches its optimal parameterization with respect to a given set of objectives. The functional class should be tailored to the problem at hand and its selection is crucial, as it determines the search space within which solutions can be found. In MOMDPs problems, a different objective tradeoff determines a different fitness landscape, requiring a tradeoff-dynamic functional class selection. Yet, in state-of-the-art applications, the policy class is generally selected a priori and kept constant across the multidimensional objective space. In this work, we present a novel policy search routine called neuro-evolutionary multiobjective DPS (NEMODPS), which extends the DPS problem formulation to conjunctively search the policy functional class and its parameterization in a hyperspace containing policy architectures and coefficients. NEMODPS begins with a population of minimally structured approximating networks and progressively builds more sophisticated architectures by topological and parametrical mutation and crossover, and selection of the fittest individuals concerning multiple objectives. We tested NEMODPS for the problem of designing the control policy of a multipurpose water system. Numerical results show that the tradeoff-dynamic structural and parametrical policy search of NEMODPS is consistent across multiple runs, and outperforms the solutions designed via traditional DPS with predefined policy topologies.

Online Microservice Orchestration for IoT via Multiobjective Deep Reinforcement Learning

By providing loosely coupled, lightweight, and independent services, the microservice architecture is promising for large-scale and complex service provision requirements in the Internet of Things (IoT). However, it requires more fine-grained resource management and orchestration for service provision. Most of the existing microservice orchestration solutions are based on those designed for the traditional cloud. They can only provide coarse-grained resource allocation using possibly conflicting weighted objectives. In this article, we present a fine-grained microservice orchestration approach to provide services online for dynamic requests of IoT applications. By using a fine-grained resource model of energy cost and service end-to-end response time of orchestrated microservices, we formulate the microservice orchestration problem as a multiobjective Markov decision process. We then propose a multiobjective optimization solution based on deep reinforcement learning (DRL) to simultaneously reduce energy consumption and response time. Through extensive experiments, our proposed algorithm presents significant performance results than the state of the art. To the best of our knowledge, this is the first work that addresses microservice orchestration using DRL for multiple conflicting objectives.

Mobile Communications, Computing, and Caching Resources Allocation for Diverse Services via Multi-Objective Proximal Policy Optimization

Mobile services are becoming more diverse, making them have different demands on communications, computing, and caching (3C) resources in mobile systems. Unlike the traditional work that considers only one type of service, this paper designs a unified framework to characterize the different kinds of services, and jointly optimizes the 3C resources of the base station (BS) and mobile devices to provide differentiated quality of service (QoS) for diverse services. In the proposed framework, we model the task required by the mobile device to be generated at the BS, the mobile device, or both of them, which means the requested tasks are served through different paths, consuming different bandwidth, computing and caching resources. Since diverse services have different QoS, we formulate a multi-objective programming (MOP) to optimize the allocation of the 3C resources for minimizing the total delay while maximizing the number of executed tasks requested by the mobile devices. We transform the MOP problem as a multi-objective Markov decision process (MO-MDP) and design a multi-objective proximal policy optimization (MO-PPO) algorithm to solve the MO-MDP. The proposed MO-PPO first trains two sub-policies separately for the two objectives, and then combines them to search for Pareto dominating solutions. By alternately perform the separate training and the combination, we can finally obtain a set of Pareto optimal solutions and the corresponding Pareto front. Simulation results show that the proposed MO-PPO outperforms traditional methods in finding a higher-quality set of Pareto optimal solutions and can more appropriately allocate 3C resources to different types of services.

Avoiding Negative Side Effects of Autonomous Systems in the Open World

Autonomous systems that operate in the open world often use incomplete models of their environment. Model incompleteness is inevitable due to the practical limitations in precise model specification and data collection about open-world environments. Due to the limited fidelity of the model, agent actions may produce negative side effects (NSEs) when deployed. Negative side effects are undesirable, unmodeled effects of agent actions on the environment. NSEs are inherently challenging to identify at design time and may affect the reliability, usability and safety of the system. We present two complementary approaches to mitigate the NSE via: (1) learning from feedback, and (2) environment shaping. The solution approaches target settings with different assumptions and agent responsibilities. In learning from feedback, the agent learns a penalty function associated with a NSE. We investigate the efficiency of different feedback mechanisms, including human feedback and autonomous exploration. The problem is formulated as a multi-objective Markov decision process such that optimizing the agent’s assigned task is prioritized over mitigating NSE. A slack parameter denotes the maximum allowed deviation from the optimal expected reward for the agent’s task in order to mitigate NSE. In environment shaping, we examine how a human can assist an agent, beyond providing feedback, and utilize their broader scope of knowledge to mitigate the impacts of NSE. We formulate the problem as a human-agent collaboration with decoupled objectives. The agent optimizes its assigned task and may produce NSE during its operation. The human assists the agent by performing modest reconfigurations of the environment so as to mitigate the impacts of NSE, without affecting the agent’s ability to complete its assigned task. We present an algorithm for shaping and analyze its properties. Empirical evaluations demonstrate the trade-offs in the performance of different approaches in mitigating NSE in different settings.

Multi-objective dynamic programming with limited precision

This paper addresses the problem of approximating the set of all solutions for Multi-objective Markov Decision Processes. We show that in the vast majority of interesting cases, the number of solutions is exponential or even infinite. In order to overcome this difficulty we propose to approximate the set of all solutions by means of a limited precision approach based on White’s multi-objective value-iteration dynamic programming algorithm. We prove that the number of calculated solutions is tractable and show experimentally that the solutions obtained are a good approximation of the true Pareto front.

Offloading dependent tasks in multi-access edge computing: A multi-objective reinforcement learning approach

This paper studies the problem of offloading an application consisting of dependent tasks in multi-access edge computing (MEC). This problem is challenging because multiple conflicting objectives exist, e.g., the completion time, energy consumption, and computation overhead should be optimized simultaneously. Recently, some reinforcement learning (RL) based methods have been proposed to address the problem. However, these methods, called single-objective RLs (SORLs), define the user utility as a linear scalarization. The conflict between objectives has been ignored. This paper formulates a multi-objective optimization problem to simultaneously minimize the application completion time, energy consumption of the mobile device, and usage charge for edge computing, subject to dependency constraints. Moreover, the relative importance (preferences) between the objectives may change over time in MEC, making it quite challenging for traditional SORLs to handle. To overcome this, we first model a multi-objective Markov decision process, where the scalar reward is extended to a vector-valued reward. Each element in the reward corresponds to one of the objectives. Then, we propose an improved multi-objective reinforcement learning (MORL) algorithm, where a tournament selection scheme is designed to select important preferences to effectively maintain previously learned policies. The simulation results demonstrate that the proposed algorithm obtains a good tradeoff between three objectives and has significant performance improvement compared with a number of existing algorithms.

Intrinsically Motivated Hierarchical Policy Learning in Multiobjective Markov Decision Processes

The multiobjective Markov decision processes (MOMDPs) are sequential decision-making problems that involve multiple conflicting reward functions that cannot be optimized simultaneously without a compromise. This type of problem cannot be solved by a single optimal policy as in the conventional case. Alternatively, multiobjective reinforcement learning (RL) methods evolve a coverage set of optimal policies that can satisfy all possible preferences in solving the problem. However, many of these methods cannot generalize their coverage sets to work in the nonstationary environments. In these environments, the parameters of the state transition and reward distribution vary over time. This limitation results in significant performance degradation for the evolved policy sets. In order to overcome this limitation, there is a need to learn a generic skillset that can bootstrap the evolution of the policy coverage set for each shift in the environment dynamics, therefore, it can facilitate a continuous learning process. In this article, intrinsically motivated RL (IMRL) has been successfully deployed to evolve generic skillsets for learning hierarchical policy to solve the MOMDPs. We propose a novel dual-phase IMRL method to address this limitation. In the first phase, a generic set of skills is learned, while in the second phase, this set is used to bootstrap policy coverage sets for each shift in the environment dynamics. We show experimentally that the proposed method significantly outperforms the state-of-the-art multiobjective reinforcement methods on a dynamic robotics environment.

A fuzzy-inference-based reinforcement learning method of overtaking decision making for automated vehicles

Intelligent decision control is one key issue of automated vehicles. Complex dynamic traffic flow and multi-requirement of passengers including vehicle safety, comfort, vehicle efficiency bring about tremendous challenges to vehicle decision making. Overtaking maneuver is a demanding task due to its large potential of traffic collision. Therefore, this paper proposes a fuzzy-inference-based reinforcement learning (FIRL) approach of autonomous overtaking decision making. Firstly, the problem of overtaking is formulated as a multi-objective Markov decision process (MDP) considering vehicle safety, driving comfort, and vehicle efficiency. Secondly, a temporal difference learning based on dynamic fuzzy (DF-TDL) is presented to learn optimized policies for autonomous overtaking decision making. Fuzzy inference is introduced to deal with continuous states and boost learning process. The RL algorithm decides whether to overtake or not based on the learned policies. Then, the automated vehicle executes local path planning and tracking. Furthermore, a simulation platform based on simulation of urban mobility (SUMO) is established to generate the random training data, that is, various traffic flows for algorithm iterative learning and validate the proposed method, extensive test results demonstrate the effectiveness of the overtaking decision-making method.

The impact of environmental stochasticity on value-based multiobjective reinforcement learning

We report a previously unidentified issue with model-free, value-based approaches to multiobjective reinforcement learning in the context of environments with stochastic state transitions. An example multiobjective Markov Decision Process (MOMDP) is used to demonstrate that under such conditions these approaches may be unable to discover the policy which maximises the Scalarised Expected Return, and in fact may converge to a Pareto-dominated solution. We discuss several alternative methods which may be more suitable for maximising SER in MOMDPs with stochastic transitions.

Learning adversarial attack policies through multi-objective reinforcement learning

Deep Reinforcement Learning has shown promising results in learning policies for complex sequential decision-making tasks. However, different adversarial attack strategies have revealed the weakness of these policies to perturbations to their observations. Most of these attacks have been built on existing adversarial example crafting techniques used to fool classifiers, where an adversarial attack is considered a success if it makes the classifier outputs any wrong class. The major drawback of these approaches when applied to decision-making tasks is that they are blind for long-term goals. In contrast, this paper suggests that it is more appropriate to view the attack process as a sequential optimization problem, with the aim of learning a sequence of attacks, where the attacker must consider the long-term effects of each attack. In this paper, we propose that such an attack policy must be learned with two objectives in view. On the one hand, the attack must pursue the maximum performance loss of the attacked policy. On the other hand, it also should minimize the cost of the attacks. Therefore, in this paper we propose a novel modelization of the process of learning an attack policy as a Multi-objective Markov Decision Process with two objectives: maximizing the performance loss of the attacked policy and minimizing the cost of the attacks. We also reveal the conflicting nature of these two objectives and use a Multi-objective Reinforcement Learning algorithm to draw the Pareto fronts for four well-known tasks: the GridWorld, the Cartpole, the Mountain car and the Breakout.