Multi-objective Reward Function Research Articles

Deep Reinforcement Learning Control for Pulsed Power Load Online Deployment in DC Shipboard Integrated Power System

Online deployment of pulsed power load (PPL) is one of the most challenging issues in DC shipboard integrated power systems (SIPSs), which leads to a multi-objective optimal control problem subject to various constraints in this paper. Since traditional model-based methods face difficulties in designing the optimal control policy and are prone to model inaccuracy and parameter uncertainty, there is an urgent need for a model-free and also high-performance control approach. Thus, a deep reinforcement learning (DRL) optimal control, which employs the twin-delayed deep deterministic policy gradient (TD3) algorithm, is presented in this paper. The DRL optimal control adopts a stack-based state observation technique to enhance learning and control performance, and it uses a multi-objective reward function design to signify the overall dynamic performance. Besides achieving the safe and fast online deployment of PPL, it also fulfills the regulation of DC bus voltage and the proportional current sharing among distributed generations (DGs). Moreover, the DRL control has an advantage in handling the ramp rate constraints of SIPS. The optimal control satisfying ramp rate constraints can be obtained through a deep learning process. The performance of the proposed DRL control is validated by case studies considering different load conditions.

Reinforcement Learning for Neural Architecture Search in Hyperspectral Unmixing

In this letter, a novel neural architecture search (NAS) method based on reinforcement learning, called RLNAS, is devised to realize the automatic architecture design in the field of hyperspectral unmixing (HU). This method first train the search network in the constructed self-supervised datasets based on hyperspectral images. The block-based searching and weight-sharing strategies are then introduced to reduce the computational cost in the training phase. The final optimal architecture is obtained by optimizing the multi-objective reward function to balance the trade-off between accuracy and computational efficiency. Compared with the state-of-the-art unmixing algorithms, the proposed RLNAS method can yield better unmixing results on synthetic and real hyperspectral datasets, which verifies its effectiveness and superiority. In addition, the proposed method offers promising potential of the NAS for HU.

A Comprehensive Analysis of Reward Function for Adaptive Traffic Signal Control

Adaptive traffic control systems (ATCS) can play an important role to reduce traffic congestion in urban areas. The main challenge for ATSC is to determine the proper signal timing. Recently, Deep Reinforcement learning (DRL) is used to determine proper signal timing. However, the success of the DRL algorithm depends on the appropriate reward function design. There exist various reward functions for ATSC in the existing research. In this research, a comprehensive analysis of the widely used reward function is presented. The pros and cons of various reward algorithms are discussed and experimental analysis shows that multi-objective reward function enhances the performance of ATSC.

Applying reinforcement learning to plan manufacturing material handling

Applying machine learning methods to improve the efficiency of complex manufacturing processes, such as material handling, can be challenging. The interconnectedness of the multiple components that make up real-world manufacturing processes and the typically very large number of variables required to specify procedures and plans within them combine to make it very difficult to map the details of such processes to a formal mathematical representation suitable for conventional optimization methods. Instead, in this work reinforcement learning was applied to produce increasingly efficient plans for material handling in representative manufacturing facilities. Doing so included defining a formal representation of a realistically complex material handling plan, specifying a set of suitable plan change operators as reinforcement learning actions, implementing a simulation-based multi-objective reward function that considers multiple components of material handling costs, and abstracting the many possible material handling plans into a state set small enough to enable reinforcement learning. Experimentation with multiple material handling plans on two separate factory layouts indicated that reinforcement learning could consistently reduce the cost of material handling. This work demonstrates the applicability of reinforcement learning with a multi-objective reward function to realistically complex material handling processes.

I-SEE: Intelligent, Secure, and Energy-Efficient Techniques for Medical Data Transmission Using Deep Reinforcement Learning

The rapid evolution of remote health monitoring applications is foreseen to be a crucial solution for facing an unpredictable health crisis and improving the quality of life. However, such applications come with many challenges, including: the transmission of a large amount of private medical data and the limited power budget for battery-operated devices. Thus, this article proposes an intelligent, secure, and energy-efficient (I-SEE) framework for secure and energy-efficient medical data transmission, leveraging the potential of physical-layer security. In particular, we incorporate a practical secrecy metric, namely, the secrecy outage probability (SOP), along with the adaptive compression at the edge for providing a secure solution for health monitoring applications. In the proposed framework, we first formulate an optimization problem that maximizes the energy efficiency, while maintaining quality-of-service constraints of the health application. Second, we propose a deep reinforcement learning process that obtains the optimal strategy for secure data transmission. Specifically, a multiobjective reward function is defined to optimize energy efficiency and distortion, resulting from the compression scheme. Then, a deep deterministic policy gradients (DDPGs) algorithm, named Static-DDPG is proposed to solve our problem efficiently. Third, the problem is extended to consider the battery lifetime maximization with varying channel conditions. Indeed, a Dynamic-DDPG algorithm is proposed in order to allow the edge to adapt to the environment dynamics while maximizing its battery lifetime. The conducted simulations validate the efficiency of the proposed algorithms in terms of finding the optimal policy that addresses the tradeoff between the considered conflicting objectives, along with the battery lifetime maximization

DeepGraphMolGen, a multi-objective, computational strategy for generating molecules with desirable properties: a graph convolution and reinforcement learning approach

We address the problem of generating novel molecules with desired interaction properties as a multi-objective optimization problem. Interaction binding models are learned from binding data using graph convolution networks (GCNs). Since the experimentally obtained property scores are recognised as having potentially gross errors, we adopted a robust loss for the model. Combinations of these terms, including drug likeness and synthetic accessibility, are then optimized using reinforcement learning based on a graph convolution policy approach. Some of the molecules generated, while legitimate chemically, can have excellent drug-likeness scores but appear unusual. We provide an example based on the binding potency of small molecules to dopamine transporters. We extend our method successfully to use a multi-objective reward function, in this case for generating novel molecules that bind with dopamine transporters but not with those for norepinephrine. Our method should be generally applicable to the generation in silico of molecules with desirable properties.

A Decision-Making Strategy for Vehicle Autonomous Braking in Emergency via Deep Reinforcement Learning

Autonomous braking through vehicle precise decision-making and control to reduce accidents is a key issue, especially in the early diffusion phase of autonomous vehicle development. This paper proposes a deep reinforcement learning (DRL)-based autonomous braking decision-making strategy in an emergency situation. Three key influencing factors, including efficiency, accuracy and passengers' comfort, are fully considered and satisfied by the proposed strategy. First, the vehicle lane-changing process and the braking process are analyzed in detail, which include the critical factors in the design of the autonomous braking strategy. Second, we propose a DRL process that determines the optimal strategy for autonomous braking. Particularly, a multi-objective reward function is designed, which can compromise the rewards achieved of different brake moments, the degree of the accident, and the comfort of the passenger. Third, a typical actor-critic (AC) algorithm named deep deterministic policy gradient (DDPG) is adopted for solving the autonomous braking problem, which can improve the efficiency of the optimal strategy and be stable in continuous control tasks. Once the strategy is well trained, the vehicle can automatically take optimal braking behavior in an emergency to improve driving safety. Extensive simulations validate the effectiveness and efficiency of our proposal in terms of learning effectiveness, decision-making accuracy and driving safety.

Automatic Virtual Network Embedding: A Deep Reinforcement Learning Approach With Graph Convolutional Networks

Virtual network embedding arranges virtual network services onto substrate network components. The performance of embedding algorithms determines the effectiveness and efficiency of a virtualized network, making it a critical part of the network virtualization technology. To achieve better performance, the algorithm needs to automatically detect the network status which is complicated and changes in a time-varying manner, and to dynamically provide solutions that can best fit the current network status. However, most existing algorithms fail to provide automatic embedding solutions in an acceptable running time. In this paper, we combine deep reinforcement learning with a novel neural network structure based on graph convolutional networks, and propose a new and efficient algorithm for automatic virtual network embedding. In addition, a parallel reinforcement learning framework is used in training along with a newly-designed multi-objective reward function, which has proven beneficial to the proposed algorithm for automatic embedding of virtual networks. Extensive simulation results under different scenarios show that our algorithm achieves best performance on most metrics compared with the existing state-of-the-art solutions, with upto 39.6% and 70.6% improvement on acceptance ratio and average revenue, respectively. Moreover, the results also demonstrate that the proposed solution possesses good robustness.

Unexpected Collision Avoidance Driving Strategy Using Deep Reinforcement Learning

In this paper, we generated intelligent self-driving policies that minimize the injury severity in unexpected traffic signal violation scenarios at an intersection using the deep reinforcement learning. We provided guidance on reward engineering in terms of the multiplicity of objective function. We used a deep deterministic policy gradient method in the simulated environment to train self-driving agents. We designed two agents, one with a single-objective reward function of collision avoidance and the other with a multi-objective reward function of both collision avoidance and goal-approaching. We evaluated their performances by comparing the percentages of collision avoidance and the average injury severity against those of human drivers and an autonomous emergency braking (AEB) system. The percentage of collision avoidance of our agents were 78.89% higher than human drivers and 84.70% higher than the AEB system. The average injury severity score of our agents were only 8.92% of human drivers and 6.25% of the AEB system.

Parameterized Batch Reinforcement Learning for Longitudinal Control of Autonomous Land Vehicles

This paper presents a parameterized batch reinforcement learning algorithm for near-optimal longitudinal control of autonomous land vehicles (ALVs). The proposed approach uses an actor-critic architecture, where parameterized feature vectors based on kernels are learned from collected samples for approximating the value functions and policies. One difference between the parameterized batch actor-critic (PBAC) algorithm and previous actor-critic learning approaches is that the critic and actor in PBAC share the same linear features, which has been theoretically proved to be a beneficial property for the convergence of actor-critic learning approaches. In order to obtain better learning efficiency, least-squares-based batch updating rules are designed for the critic and actor, respectively. Based on the PBAC learning algorithm, a data-driven longitudinal control method is presented for ALVs to obtain near-optimal control policies which adaptively tune the fuel/brake control signals to track different speeds. A multiobjective reward function is designed so that both tracking precision and driving smoothness are considered. Extensive experiments were conducted on a real ALV platform while driving on flat, slippery, sloping, and bumpy roads. The experimental results illustrate the superiority of the PBAC-based self-learning controller over conventional longitudinal control methods such as proportional-integral (PI) control and learning-based PI control.

Behavior sequencing based on demonstrations: a case of a humanoid opening a door while walking

There is neuroscientific evidence to suggest that imitation between humans is goal-directed. Therefore, when performing multiple tasks, we internally define an unknown optimal policy to satisfy multiple goals. This work presents a method to transfer a complex behavior composed by a sequence of multiple tasks from a human demonstrator to a humanoid robot. We defined a multi-objective reward function as a measurement of the goal optimality for both human and robot, which is defined in each subtask of the global behavior. We optimize a sequential policy to generate whole-body movements for the robot that produces a reward profile which is compared and matched with the human reward profile, producing an imitative behavior. Furthermore, we can search in the proximity of the solution space to improve the reward profile and innovate a new solution, which is more beneficial for the humanoid. Experiments were carried out in a real humanoid robot.

Full-Body Postural Control of a Humanoid Robot with Both Imitation Learning and Skill Innovation

In this paper, we present a novel methodology to obtain imitative and innovative postural movements in a humanoid based on human demonstrations in a different kinematic scale. We collected motion data from a group of human participants standing up from a chair. Modeling the human as an actuated 3-link kinematic chain, and by defining a multi-objective reward function of zero moment point and joint torques to represent the stability and effort, we computed reward profiles for each demonstration. Since individual reward profiles show variability across demonstrating trials, the underlying state transition probabilities were modeled using a Markov chain. Based on the argument that the reward profiles of the robot should show the same temporal structure of those of the human, we used differential evolution to compute a trajectory that fits all humanoid constraints and minimizes the difference between the robot reward profile and the predicted profile if the robot imitates the human. Therefore, robotic imitation involves developing a policy that results in a temporal reward structure, matching that of a group of human demonstrators across an array of demonstrations. Skill innovation was achieved by optimizing a signed reward error after imitation was achieved. Experimental results using the humanoid HOAP-3 are shown.