Multi-objective Reward Function Research Articles

Multi-agent DRL-based Multipath Scheduling for Video Streaming with QUIC

The popularization of video streaming brings challenges in satisfying diverse Quality of Service (QoS) requirements. The multipath extension of the Quick UDP Internet Connection (QUIC) protocol, also called MPQUIC, has the potential to improve video streaming performance with multiple simultaneously transmitting paths. The multipath scheduler of MPQUIC determines how to distribute the packets onto different paths. However, while applying current multipath schedulers into MPQUIC, our experimental results show that they fail to adapt to various receive buffer sizes of different devices and comprehensive QoS requirements of video streaming. These problems are especially severe under heterogeneous and dynamic network environments. To tackle these problems, we propose MARS, a Multi-agent deep Reinforcement learning (MADRL)-based Multipath QUIC Scheduler, which is able to promptly adapt to dynamic network environments. It exploits the MADRL method to learn a neural network for each path and generate scheduling policy. Besides, it introduces a novel multi-objective reward function that takes out-of-order queue size and different QoS metrics into consideration to realize adaptive scheduling optimization. We implement MARS in an MPQUIC prototype and deploy in Dynamic Adaptive Streaming over HTTP system. Then, we compare it with the state-of-the-art multipath schedulers in both emulated and real-world networks. Experimental results show that MARS outperforms the other schedulers with better adaptive capability regarding the receive buffer sizes and QoS.

Cooperative traffic optimization with multi-agent reinforcement learning and evolutionary strategy: Bridging the gap between micro and macro traffic control

The emergence of connected and autonomous vehicles (CAVs) holds promise for fine-grained traffic control. However, due to the longevity of future mixed traffic scenarios, there is a need for an in-depth exploration of integrating the microscopic speed control of CAVs with the macroscopic variable speed limit (VSL) of human-driven vehicles (HDVs). This paper proposes a Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME) framework, which combines microscopic CAV control with macroscopic VSL control. The framework incorporates a Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework for intelligent decision-making by microscopic-level vehicles. Additionally, an evolutionary strategy is developed to design the VSL network architecture, enabling macroscopic level real-time speed limit adjustments based on infrastructure. A multi-objective reward function is proposed to optimize both micro and macro efficiency and safety, accounting for both vehicle behavior and traffic flow. Experiments on the designed Bottleneck traffic scenarios show that the proposed approach, CTO-ME, is able to achieve superior performance and outperforms other baselines in terms of traffic throughput, average speed, and safety. Specifically, CTO-ME enhances average velocity by 37%, increases overall throughput by 309%, and raises arrival ratio by 70% than traditional Intelligent Driver Model (IDM).

An intelligent schedule maintenance method for hydrogen fuel cell vehicles based on deep reinforcement learning considering dynamic reliability

Schedule maintenance activities for hydrogen fuel cell vehicles are aimed to keep them in better condition in long-term operation. In this paper, we develop a reinforcement learning based schedule maintenance strategy, which is used to comprehensively consider the balance between safety and maintenance cost, so as to find the optimal maintenance strategy. A multi-level framework is established with remaining useful life of key components, fault tree analysis, and logistics cost model to generate an exploration environment that can express the operational stability, the failure rate of components, and the cost of repair and storage. The trained agent combines a deep neural network to explore the optimal strategy under the dynamic reliability of key components. Finite steps are the constraints. Safety rates, maintenance cost, and episode of operation are incorporated into a multi-objective reward function. The hydrogen supply circuit of fuel cell vehicle is simulated via Python. The trained agent is compared with traditional time-based schedule maintenance strategy and corrective maintenance strategy. The results show that the reinforcement learning based schedule maintenance agent has optimized the total reward, cost control and accident rate by 77%, 59% and 5%, respectively, compared with the traditional time-based schedule maintenance strategy. In the safe operation management of fuel cell vehicles, efficient and stable decision-making ability has been achieved.

Deep reinforcement learning algorithm based ramp merging decision model

On-ramp merge is a complex traffic scenario in autonomous driving. Because of the uncertainty of the driving environment, most rule-based models cannot solve such a problem. This paper designs a ramp merging decision model based on deep deterministic policy gradient algorithm (DDPG) to solve the vehicle merging problem. To address the problems of slow algorithm merging and poor robustness of previous deep reinforcement learning algorithms in the field of intelligent vehicle ramp merging leading to the low success rate of intelligent vehicle merging, first, we introduce a simple recurrent unit (SRU) for extracting intelligent vehicle states and environment features and use the DDPG algorithm for intelligent vehicle decision making. Second, the experience playback pool of DDPG algorithm is improved by using priority sampling instead of uniform sampling. Finally, a multi-objective reward function is set up during training, considering factors such as safety and efficiency. The simulation experiments show that the improved algorithm improves the merging speed of the model, reduces the collision rate, and enables the vehicle to make more reasonable decisions. In addition, the superiority of the method is demonstrated by comparing with the advanced method.

Empty container repositioning problem using a reinforcement learning framework with multi-weight adaptive reward function

ABSTRACT In logistics networks, empty container congestion and scarcity often stem from trade imbalance and supply-demand mismatch. This paper focuses on the problem of empty container repositioning in maritime logistics and proposes a reinforcement learning framework that integrates a self-adaptive mechanism for adjusting the weights of a multi-objective reward function. The objective is to enhance container utilization and reduce scheduling costs. By reviewing the development of the empty container repositioning problem and analysing the advantages of using reinforcement learning to address the temporal and spatial complexity, the problem is modelled as a Markov decision process and tackled using reinforcement learning techniques. To achieve the optimization objectives, which involve reducing resource shortages at various locations and minimizing resource repositioning costs, a multi-objective reward function is introduced to capture the mutually constrained preferences. The weights of the reward function are dynamically adjusted to account for the potential time-varying preferences of the agent, mitigating the issue of poor generalization performance associated with fixed-weight reward functions. Comparative experimental analysis against conventional reinforcement learning algorithms demonstrates the superior performance of the proposed approach in problem solving. Based on the results and practical requirements of the case study, relevant recommendations for addressing empty container repositioning are presented.

Optimal online energy management strategy of a fuel cell hybrid bus via reinforcement learning

An energy management strategy (EMS) based on reinforcement learning is proposed in this study to enhance the fuel economy and durability of a fuel cell hybrid bus (FCHB). Firstly, a comprehensive powertrain system model for the FCHB is established, mainly including the FCHB’s power balance, fuel cell system (FCS) efficiency, and aging models. Secondly, the state–action space, state transition probability matrix (TPM), and multi-objective reward function of Q-learning algorithm are designed to improve the fuel economy and the durability of power sources. The FCHB’s demand power and battery state of charge (SOC) serve as the state variables and the FCS output power is used as the action variable. Using the demonstration FCHB data, a state TPM is created to represent the overall operation. Finally, an EMS employing Q-learning is formulated to optimize the fuel economy of FCHB, maintain battery SOC, suppress FCS power fluctuations, and enhance FCS lifetime. The proposed EMS is tested and verified through hardware-in-the-loop (HIL) tests. The simulation results demonstrate the effectiveness of the proposed strategy. Compared to a rule-based EMS, the Q-learning-based EMS can improve the energy economy by 7.8%. Furthermore, it is only a 3.7% difference to the best energy economy under dynamic optimization, while effectively reducing the decline and enhancing the durability of the FCS.

A health-aware energy management strategy for fuel cell hybrid electric UAVs based on safe reinforcement learning

Energy management strategies (EMSs) are crucial for the hydrogen economy and energy component lifetimes of fuel cell hybrid electric unmanned aerial vehicles (UAVs). Reinforcement learning (RL)-based schemes have been a hotspot for EMSs, but most of RL-based EMSs focus on the energy-saving performance and rarely consider energy component durability and safe exploration. This paper proposes a health-aware energy management strategy based on a safe RL framework to minimize the overall flight cost and achieve safe operation of UAVs. In this framework, a universal three-dimensional environment that integrates the UAV kinematics and dynamics model is developed. In addition, wind disturbances and random loading of the mission payload during flight are considered for robust training. The energy management problem is formulated as a constrained Markov decision process, where both hydrogen consumption and energy component degradation are incorporated in the multi-objective reward function. A safety optimizer is then designed to satisfy operation constraints by correcting the action through analytical optimization. The results indicate that the safety of the explored action is guaranteed, maintaining zero constraint violations in both training and real-time control scenarios. Compared with other RL-based methods, the proposed method had better convergence capability and reduced the training time. Furthermore, the simulation showed that the proposed method can reduce the total flight cost and fuel cell degradation by 14.6% and 15.3%, respectively, compared with the online benchmark method.

Energy-efficient adaptive cruise control system with acceleration prediction via long short-term memory

An energy-efficient adaptive cruise control (ACC) system that predicts the preceding state is proposed to meet the demands of driving in different states and working conditions. The long short-term memory (LSTM) network predicts the trajectory of the preceding vehicle’s future acceleration, allowing for an adaptive time headway that takes acceleration into account. The acceleration of the preceding vehicle is also incorporated as an augmented state in the deep deterministic policy gradient (DDPG) algorithm, resulting in a combined algorithm called prediction deep deterministic policy gradient (PDDPG). A multi-objective reward function is constructed based on human driving data to evaluate the performance indexes of vehicle longitudinal control, including efficiency, safety, and economy. The rules for changing the weight of each target performance under various typical cycle conditions are determined through experiments. The Carsim-based urban, suburban and highway conditions together with the Simulink vehicle dynamics model are compared with human driving data and ACC of conventional algorithms. Based on the test results, the proposed algorithm can improve fuel efficiency by 19.36% in urban driving conditions and reduce acceleration fluctuations.

Multi-user edge service orchestration based on Deep Reinforcement Learning

The fifth generation (5G) of mobile network offers a remarkable degree of flexibility to mobile operators, enabling them to provide users with effective and tailored network services. Software Defined Networking (SDN), Network Function Virtualization (NFV), and edge computing have given the operator the opportunity to easily bring computational capacity to the edge and to support latency-sensitive services. While 5G standards have defined the technological and architectural frameworks to orchestrate services, finding effective resources management and QoS optimization policies is still an open research issue.In this paper, we propose an online orchestration methodology for a multi-user edge service. The orchestrator goal is to simultaneously maximize the QoS, and minimize the amount of resources needed.We provide a mathematical formulation to compute an optimal offline policy and derive an online approach based on a model-free Deep Reinforcement Learning (DRL) framework. As a novel feature, the DRL agent action is modeled as a parametric combinatorial problem. A tailored multi-objective reward function leads the agent towards an effective choice of parameters for such a model. Our models are built, trained and fine-tuned by exploiting real data.Extensive simulations in diverse scenarios show that our DRL online approach produces solutions with small gaps to the optimal offline ones, enabling the operator to both save resources and grant the users an adequate QoS level.

A Discrete Soft Actor-Critic Decision-Making Strategy With Sample Filter for Freeway Autonomous Driving

Autonomous driving is a promising technology to reduce traffic accidents and improve driving efficiency. Although significant progress has been achieved, existing decision-making systems of autonomous vehicle still cannot meet the safety and driving efficiency requirements in highly dynamic environments. In this work, we design a discrete decision-making strategy based on the discrete soft actor-critic with sample filter algorithm (DSAC-SF) to improve driving efficiency and safety on freeways with dynamics traffic. Specifically, we first propose a sample filter method for discrete soft actor-critic, which improves the sample efficiency and stability of the algorithm via enhancing the utilization of effective samples. Subsequently, we construct the discrete decision-making strategy for autonomous driving based on the DSAC-SF algorithm, and further design the area observation method and the multi-objective reward function to improve the driving safety and efficiency. Finally, we carry out comparison and ablation experiments on the the scalable multi-agent reinforcement learning training school (SMARTS) simulation environment. Experimental results indicate that our strategy obtains a high success rate and a fast vehicle speed in the decision-making tasks on freeways. Moreover, our DSAC-SF algorithm also achieves improved performance in training efficiency and stability compared to the commonly used discrete reinforcement learning algorithm.

A Deep Reinforcement Learning Based Hierarchical Eco-Driving Strategy for Connected and Automated HEVs

The advancement in vehicle connectivity and autonomy has fostered the development of eco-driving technology, aimed at optimizing driving behaviors to reduce vehicle energy consumption. This study proposed a real-time deep reinforcement learning based hierarchical eco-driving control strategy to optimally control a connected and automated hybrid electric vehicle in a traffic scenario with multiple constraints. The speed optimization layer of the proposed strategy employs a twin-delayed deep deterministic policy gradient (TD3) based reference speed planning strategy to compute the optimized speed, based on a pre-trained optimal policy and observed environmental states. Specifically, to learn the optimal policy, a multi-objective reward function is designed that integrates fuel consumption reward and shaping reward involving car-following and road speed limit. Additionally, a rule-based competition-decision model is embedded within the speed optimization layer to ensure compliance with traffic light rules. In the vehicle controller layer, a real-time controller is implemented to specify appropriate actuator variables for the hybrid powertrain to track the reference speed and conduct energy management control. Simulation results show that the proposed TD3 based eco-driving strategy achieves remarkable energy saving performance by optimizing the speed. Besides, the proposed eco-driving strategy is capable of satisfying the constraint of diverse traffic scenarios, including car-following and traffic light, while also being computationally lightweight.

Integrated attitude and landing control for quadruped robots in asteroid landing mission scenarios using reinforcement learning

In this investigation, the integrated attitude and landing control scheme for quadruped robots in asteroid landing mission scenario is discussed. Compared with the gravitational field environment of the Earth, the gravitational field near most asteroids has a smaller gravitational acceleration, a non-negligible horizontal acceleration component, and obvious uneven distribution characteristics. In this study, an integrated control method is proposed which focuses on the challenging attitude and landing control schemes of quadruped robots near asteroids by using reinforcement learning in conjunction with an auto-tuned reward function. In the proposed method, attitude adjustment and landing control are trained as a whole. By relying on the trained controller, the quadruped robot can reorient automatically to the most suitable attitude for landing according to gravity and terrain information, thus only relying on the movement of mechanical legs (and not on any additional actuators like reaction wheels) during the entire process. To solve the problem of sparse reward in the process of multi-objective reinforcement learning, an autotuning method of a multi-objective reward function is proposed to improve the training speed. The effectiveness of the proposed landing control method of a quadruped robot is verified near irregular rod-shaped asteroid 216 Kleopatra. The numerical simulation results show that the quadruped robot can adjust reliably its attitude and finally land on irregular terrain without floating and escaping again, even when the gravitational acceleration is unknown and subject to large horizontal components.

Twin delayed deep deterministic policy gradient-based deep reinforcement learning for energy management of fuel cell vehicle integrating durability information of powertrain

Deep reinforcement learning (DRL)-based energy management strategy (EMS) is attractive for fuel cell vehicle (FCV). Nevertheless, the fuel economy and lifespan durability of proton exchange membrane fuel cell (PEMFC) stack and lithium-ion battery (LIB) may not be synchronously optimized since transient degradation variations of PEMFC stack and LIB are not generally regarded for DRL-based EMSs. Furthermore, the inappropriate action space and the overestimated value function of DRL can lead to suboptimal EMS for on-line control. To this end, the objective of this research endeavors to formulate a twin delayed deep deterministic policy gradient (TD3)-based EMS integrating durability information of PEMFC stack and LIB, which can interact with the vehicle operating states to continuously control the hybrid powertrain and limit the overestimation of DRL value function for ensuring maximum multi-objective reward at each moment. Unlike traditional DRL-based EMSs, the multi-objective reward function for this study is enlarged to incorporate the hydrogen consumption, state of charge (SOC)-sustaining penalty and transient lifespan degradation information of PEMFC stack and LIB in off-line training and on-line control. The results demonstrate that the proposed EMS can drastically lessen the training time and computational burden. Meanwhile, in contrast with deep Q-network (DQN)-based and deep deterministic policy gradient (DDPG)-based EMSs in the various real-world urban and standard driving cycles, the proposed EMS can achieve hydrogen abatement at least 9.76% and 1.07%, and slow down total powertrain degradation at least 9.11% and 2.62%, respectively.

A Decision-Making Strategy for Car Following Based on Naturalist Driving Data via Deep Reinforcement Learning.

To improve the satisfaction and acceptance of automatic driving, we propose a deep reinforcement learning (DRL)-based autonomous car-following (CF) decision-making strategy using naturalist driving data (NDD). This study examines the traits of CF behavior using 1341 pairs of CF events taken from the Next Generation Simulation (NGSIM) data. Furthermore, in order to improve the random exploration of the agent’s action, the dynamic characteristics of the speed-acceleration distribution are established in accordance with NDD. The action’s varying constraints are achieved via a normal distribution 3σ boundary point-to-fit curve. A multiobjective reward function is designed considering safety, efficiency, and comfort, according to the time headway (THW) probability density distribution. The introduction of a penalty reward in mechanical energy allows the agent to internalize negative experiences. Next, a model of agent-environment interaction for CF decision-making control is built using the deep deterministic policy gradient (DDPG) method, which can explore complicated environments. Finally, extensive simulation experiments validate the effectiveness and accuracy of our proposal, and the driving strategy is learned through real-world driving data, which is better than human data.

Exploring reward efficacy in traffic management using deep reinforcement learning in intelligent transportation system

In the last decade, substantial progress has been achieved in intelligent traffic control technologies to overcome consistent difficulties of traffic congestion and its adverse effect on smart cities. Edge computing is one such advanced progress facilitating real-time data transmission among vehicles and roadside units to mitigate congestion. An edge computing-based deep reinforcement learning system is demonstrated in this study that appropriately designs a multiobjective reward function for optimizing different objectives. The system seeks to overcome the challenge of evaluating actions with a simple numerical reward. The selection of reward functions has a significant impact on agents' ability to acquire the ideal behavior for managing multiple traffic signals in a large-scale road network. To ascertain effective reward functions, the agent is trained withusing the proximal policy optimization method in several deep neural network models, including the state-of-the-art transformer network. The system is verified using both hypothetical scenarios and real-world traffic maps. The comprehensive simulation outcomes demonstrate the potency of the suggested reward functions.

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL) has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with human-driven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision-making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic (MA2C) method is proposed with a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is designed to incorporate fuel efficiency, driving comfort, and the safety of autonomous driving. A comprehensive experimental study is made that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency, safety, and driver comfort.

Deep Reinforcement Learning Control for Pulsed Power Load Online Deployment in DC Shipboard Integrated Power System

Online deployment of pulsed power load (PPL) is one of the most challenging issues in DC shipboard integrated power systems (SIPSs), which leads to a multi-objective optimal control problem subject to various constraints in this paper. Since traditional model-based methods face difficulties in designing the optimal control policy and are prone to model inaccuracy and parameter uncertainty, there is an urgent need for a model-free and also high-performance control approach. Thus, a deep reinforcement learning (DRL) optimal control, which employs the twin-delayed deep deterministic policy gradient (TD3) algorithm, is presented in this paper. The DRL optimal control adopts a stack-based state observation technique to enhance learning and control performance, and it uses a multi-objective reward function design to signify the overall dynamic performance. Besides achieving the safe and fast online deployment of PPL, it also fulfills the regulation of DC bus voltage and the proportional current sharing among distributed generations (DGs). Moreover, the DRL control has an advantage in handling the ramp rate constraints of SIPS. The optimal control satisfying ramp rate constraints can be obtained through a deep learning process. The performance of the proposed DRL control is validated by case studies considering different load conditions.

Reinforcement Learning for Neural Architecture Search in Hyperspectral Unmixing

In this letter, a novel neural architecture search (NAS) method based on reinforcement learning, called RLNAS, is devised to realize the automatic architecture design in the field of hyperspectral unmixing (HU). This method first train the search network in the constructed self-supervised datasets based on hyperspectral images. The block-based searching and weight-sharing strategies are then introduced to reduce the computational cost in the training phase. The final optimal architecture is obtained by optimizing the multi-objective reward function to balance the trade-off between accuracy and computational efficiency. Compared with the state-of-the-art unmixing algorithms, the proposed RLNAS method can yield better unmixing results on synthetic and real hyperspectral datasets, which verifies its effectiveness and superiority. In addition, the proposed method offers promising potential of the NAS for HU.

A Comprehensive Analysis of Reward Function for Adaptive Traffic Signal Control

Adaptive traffic control systems (ATCS) can play an important role to reduce traffic congestion in urban areas. The main challenge for ATSC is to determine the proper signal timing. Recently, Deep Reinforcement learning (DRL) is used to determine proper signal timing. However, the success of the DRL algorithm depends on the appropriate reward function design. There exist various reward functions for ATSC in the existing research. In this research, a comprehensive analysis of the widely used reward function is presented. The pros and cons of various reward algorithms are discussed and experimental analysis shows that multi-objective reward function enhances the performance of ATSC.

Applying reinforcement learning to plan manufacturing material handling

Applying machine learning methods to improve the efficiency of complex manufacturing processes, such as material handling, can be challenging. The interconnectedness of the multiple components that make up real-world manufacturing processes and the typically very large number of variables required to specify procedures and plans within them combine to make it very difficult to map the details of such processes to a formal mathematical representation suitable for conventional optimization methods. Instead, in this work reinforcement learning was applied to produce increasingly efficient plans for material handling in representative manufacturing facilities. Doing so included defining a formal representation of a realistically complex material handling plan, specifying a set of suitable plan change operators as reinforcement learning actions, implementing a simulation-based multi-objective reward function that considers multiple components of material handling costs, and abstracting the many possible material handling plans into a state set small enough to enable reinforcement learning. Experimentation with multiple material handling plans on two separate factory layouts indicated that reinforcement learning could consistently reduce the cost of material handling. This work demonstrates the applicability of reinforcement learning with a multi-objective reward function to realistically complex material handling processes.