Continuous Space Problems Research Articles

Energy Management for Hybrid Energy Storage System in Electric Based on Deep Deterministic Policy Gradient

In this paper, an intelligent control system design scheme based on deep deterministic policy gradient (DDPG) algorithm is proposed for the complex continuous action space problem in the hybrid energy storage system of electric vehicles. Firstly, the basic principle and internal logic of DDPG algorithm are introduced, including key elements such as Actor-Critic architecture, experience playback, target network, reward signal, policy gradient and value function update. Then, how to apply the DDPG algorithm to the industrial control system is described in detail. The Actor network learns the optimal strategy, the Critic network evaluates the value of the state-action pair, and uses the experience playback and the target network to improve the system stability and performance. Finally, the effect of the intelligent control system based on DDPG algorithm in complex environment is verified by simulation experiments. The results show that the system can effectively optimize the control strategy, improve the response speed and stability of the system, and has a good engineering application prospect.

Active Exploration Deep Reinforcement Learning for Continuous Action Space with Forward Prediction

The application of reinforcement learning (RL) to the field of autonomous robotics has high requirements about sample efficiency, since the agent expends for interaction with the environment. One method for sample efficiency is to extract knowledge from existing samples and used to exploration. Typical RL algorithms achieve exploration using task-specific knowledge or adding exploration noise. These methods are limited to current policy improvement level and lack of long-term planning. We propose a novel active exploration deep RL algorithm for the continuous action space problem named active exploration deep reinforcement learning (AEDRL). Our method uses the Gaussian process to model dynamic model, enabling the probability description of prediction sample. Action selection is formulated as the solution of the optimization problem. Thus, the optimization objective is specifically designed for selecting samples that can minimize the uncertainty of the dynamic model. Active exploration is achieved through long-term optimized action selection. This long-term considered action exploration method is more guidance for learning. Enable intelligent agents to explore more interesting action spaces. The proposed AEDRL algorithm is evaluated on several robotic control task including classic pendulum problem and five complex articulated robots. The AEDRL can learn a controller using fewer episodes and demonstrates performance and sample efficiency.

Layerwise Quantum Deep Reinforcement Learning for Joint Optimization of UAV Trajectory and Resource Allocation

This study proposes a layerwise quantum-based deep reinforcement learning (LQ-DRL) method for optimizing continuous large space and time series problems using deep layer training. The actions in LQ-DRL are optimized using a layerwise quantum embedding that leverages the advantages of quantum computing to maximize reward and reduce training loss. Moreover, this study employs a local loss to minimize the occurrence of barren plateaus phenomena and further enhance performance. As a particular case, the proposed scheme is employed to jointly optimize: (1) UAV trajectory planning, (2) user grouping, and (3) power allocation for higher energy efficiency of a UAV as the reward. The combination of these optimized factors is referred to as action space in the presented LQ-DRL. The LQ-DRL is employed to solve the optimization problem due to its non-convexity, continuous and large action space, and time-series domain. In a practical view, LQ-DRL aims to solve the issue of energy consumption related to limited-battery energy of a UAV base station while maintaining quality-of-service (QoS) for users, by gaining maximum energy efficiency as the reward. One of real applications, as an example, LQ-DRL can be employed to maximize the energy efficiency of a UAV base station in UAV empowered disaster recovery networks scenario. The quantum circuits of layerwise quantum embedding are presented to show the practical implementation in noisy intermediate-scale quantum computers. Based on the results, LQ-DRL outperformed the classical DRL by achieving higher effective dimension, rewards, and lower learning losses. In addition, better performances were achieved using more layers.

Application of stochastic filter to three-phase nonuniform transmission lines

ABSTRACT This paper implements the stochastic filters for state and parameter estimation of nonuniform transmission lines (NTL). In general, transmission line (TL) problem is a continuous time and space problem. By taking the line loading noise into account, the TL equations become a stochastic partial differential equation (PDE) rather than a simple set of coupled finite stochastic differential equations (SDE). By transforming the spatial variables into the Fourier domain, the stochastic PDE can be transformed into an infinite sequence of SDE. After truncation to a finite set of Fourier series coefficients, it becomes a finite set of coupled linear SDE, which is the required domain in which extended Kalman filter (EKF) and unscented Kalman filter (UKF) can be applied. For state space equation of EKF and UKF, the voltage and current of periodic NTL are expanded into an infinite set of spatial harmonics. In this way, the voltage and current measurement of NTL become an eigenvalue problem. The NTL is considered as cascade of small infinite NTL and the four distributed primary parameters of the periodic NTL are expressed using Fourier series expansion. Finally, the Kalman filter (KF)-based state estimation and the EKF- and UKF-based parameter estimation have been compared with recursive least squares (RLS) method. The simulation results present the superiority of the method.

Two-Stream Fused Fuzzy Deep Neural Network for Multiagent Learning

In multiagent reinforcement learning (RL), multilayer fully connected neural network is used for value function approximation, which solves large-scale or continuous space problems. However, it is easy to fall into a local optimal and overfitting under partially observed environments. Because each agent lacks the information that plays a key role in decision making beyond the observation field. Even if communication is allowed, the received informations in communication channel have large noise due to the observations of other agents and strong uncertainty if the agent's policy is used as the communication information. To tackle this problem, two-stream fused fuzzy deep neural network (2s-FDNN) was proposed to reduce the uncertainty and noise of information in the communication channel. It is a parallel structure in which the fuzzy inference module reduces the uncertainty of information and the deep neural module reduces the noise of information. Then, we presented Fuzzy MA2C which integrates 2s-FDNN into multiagent deep RL to deal with uncertain communication informations for improving the robustness and generalization under partially observed environments. We empirically evaluate our methods in two large-scale traffic signal control environments using simulation of urban mobility (SUMO) simulator. Results demonstrate that our methods can achieve superior performance against existing RL algorithms.

On the existence and uniqueness properties of the Hoogendoorn–Bovy pedestrian flow model

This study addresses the existence and uniqueness of solutions to the Hoogendoorn–Bovy (HB) pedestrian flow model, which describes the dynamic user-optimal pedestrian flow assignment problem in continuous space and time. The HB model consists of a forward conservation law (CL) equation that governs density and a backward Hamilton–Jacobi–Bellman (HJB) equation that contains a maximum admissible speed constraint (MASC), in which the flow direction is determined by the path-choice strategy. The existence and uniqueness of solutions are significantly more difficult to determine when the HJB equation contains an MASC; however, we prove that the HB model can be formulated as a forward CL equation and backward Hamilton–Jacobi (HJ) equation in which the MASC is non-binding if suitable model parameters are selected. This model is formulated as a fixed-point problem upon the simultaneous satisfaction of both equations. To verify the existence and uniqueness results, we first demonstrate the existence and uniqueness of the solutions to the CL and HJ equations, and then show that the coupled HB model is well-posed and has a unique solution. A numerical example is presented to illustrate the properties of the HB model.

A collaborative computation and dependency-aware task offloading method for vehicular edge computing: a reinforcement learning approach

Vehicular edge computing (VEC) is emerging as a new computing paradigm to improve the quality of vehicular services and enhance the capabilities of vehicles. It enables performing tasks with low latency by deploying computing and storage resources close to vehicles. However, the traditional task offloading schemes only focus on one-shot offloading, taking less into consideration task dependency. Furthermore, the continuous action space problem during task offloading should be considered. In this paper, an efficient dependency-aware task offloading scheme for VEC with vehicle-edge-cloud collaborative computation is proposed, where subtasks can be processed locally or can be offloaded to an edge server, or a cloud server for execution. Specifically, first, the directed acyclic graph (DAG) is utilized to model the dependency of subtasks. Second, a task offloading algorithm based on Deep Deterministic Policy Gradient (DDPG) was proposed to obtain the optimal offloading strategy in a vehicle-edge-cloud environment, which efficiently solves the continuous control problem and helps reach fast convergence. Finally, extensive simulation experiments have been conducted, and the experimental results show that the proposed scheme can improve performance by about 13.62% on average against three baselines.

Research on UCAV Maneuvering Decision Method Based on Heuristic Reinforcement Learning.

With the rapid development of unmanned combat aerial vehicle (UCAV)-related technologies, UCAVs are playing an increasingly important role in military operations. It has become an inevitable trend in the development of future air combat battlefields that UCAVs complete air combat tasks independently to acquire air superiority. In this paper, the UCAV maneuver decision problem in continuous action space is studied based on the deep reinforcement learning strategy optimization method. The UCAV platform model of continuous action space was established. Focusing on the problem of insufficient exploration ability of Ornstein–Uhlenbeck (OU) exploration strategy in the deep deterministic policy gradient (DDPG) algorithm, a heuristic DDPG algorithm was proposed by introducing heuristic exploration strategy, and then a UCAV air combat maneuver decision method based on a heuristic DDPG algorithm is proposed. The superior performance of the algorithm is verified by comparison with different algorithms in the test environment, and the effectiveness of the decision method is verified by simulation of air combat tasks with different difficulty and attack modes.

Learning in continuous action space for developing high dimensional potential energy models

Reinforcement learning (RL) approaches that combine a tree search with deep learning have found remarkable success in searching exorbitantly large, albeit discrete action spaces, as in chess, Shogi and Go. Many real-world materials discovery and design applications, however, involve multi-dimensional search problems and learning domains that have continuous action spaces. Exploring high-dimensional potential energy models of materials is an example. Traditionally, these searches are time consuming (often several years for a single bulk system) and driven by human intuition and/or expertise and more recently by global/local optimization searches that have issues with convergence and/or do not scale well with the search dimensionality. Here, in a departure from discrete action and other gradient-based approaches, we introduce a RL strategy based on decision trees that incorporates modified rewards for improved exploration, efficient sampling during playouts and a “window scaling scheme" for enhanced exploitation, to enable efficient and scalable search for continuous action space problems. Using high-dimensional artificial landscapes and control RL problems, we successfully benchmark our approach against popular global optimization schemes and state of the art policy gradient methods, respectively. We demonstrate its efficacy to parameterize potential models (physics based and high-dimensional neural networks) for 54 different elemental systems across the periodic table as well as alloys. We analyze error trends across different elements in the latent space and trace their origin to elemental structural diversity and the smoothness of the element energy surface. Broadly, our RL strategy will be applicable to many other physical science problems involving search over continuous action spaces.

A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments

This paper addresses the problem of navigating decentralized multi-agent systems in partially cluttered environments and proposes a new machine-learning-based approach to solve it. On the basis of this approach, a new robust and flexible Q-learning-based model is proposed to handle a continuous space problem. As in reinforcement learning (RL) algorithms, Q-learning does not require a model of the environment. Additionally, Q-Learning (QL) has the advantages of being fast and easy to design. However, one disadvantage of QL is that it needs a massive amount of memory, and it grows exponentially with each extra feature introduced to the state space. In this research, we introduce an agent-level decentralized collision avoidance low-cost model for solving a continuous space problem in partially cluttered environments, followed by introducing a method to merge non-overlapping QL features in order to reduce its size significantly by about 70% and make it possible to solve more complicated scenarios with the same memory size. Additionally, another method is proposed for minimizing the sensory data that is used by the controller. A combination of these methods is able to handle swarm navigation low memory cost with at least18 number of robots. These methods can also be adapted for deep q-learning architectures so as to increase their approximation performance and also decrease their learning time process. Experiments reveal that the proposed method also achieves a high degree of accuracy for multi-agent systems in complex scenarios.

A possibilistic programming approach for biomass supply chain network design under hesitant fuzzy membership function estimation

The recognition of membership function by knowledge acquisition from experts is an important factor for many fuzzy mathematical programming models. Thus, the hesitant fuzzy membership function (HFMF) estimation could help users of the mathematical programming approaches to provide a powerful solution in continuous space problems. Therefore, this study proposes a possibilistic programming approach based on Bezier curve mechanism for estimating the HFMF. In the process of possibilistic programming approach, an optimization model is presented to tune the primary parameters of Bezier curve by the goal of minimizing the sum of the squared errors (SSE) between the empirical data and fitted HFMF. After that, the efficiency and applicability of the proposed approach is checked by proposing a novel mathematical model for biomass supply chain network design problem. In this case, the bio-products demand is declared as an imprecise parameter that follows from the estimated HFMF to increase the accuracy of the obtained results by addressing the uncertainty and unreliability of the information. Finally, a computational experiment and validation procedure about the biomass supply chain network design is provided to peruse the verification and validation of the proposed approaches.

Posing probability problems related to continuous and discrete sample space

In this study, it was aimed to examine the probability problems which were posed based on continuous sample space and discrete sample space. Participants of the study consisted of 48 university students selected by using the convenience sampling method. Eight tasks related to continuous and discrete variables were designed by the researcher. Eight problems posed by each student in accordance with these tasks, their reflection papers and interviews conducted with them constituted the data of the study. Problems were analysed according to appropriateness, solvability, context, applicability, language and variable type dimensions. Findings revealed that most solvable problems were in real-life context, applicable and had clear language. However, most of the unsolvable problems had missing information. Inappropriate problems either were not generated depending on the task instructions or suitable variables were not used in them. Students had difficulties particularly in posing continuous sample space problems. It was concluded that problem posing was an effective practice for students since it enhanced their knowledge about probability and helped them to gain insight into problem posing.

A Novel Behavioral Strategy for RoboCode Platform Based on Deep Q-Learning

This paper addresses a new machine learning-based behavioral strategy using the deep Q-learning algorithm for the RoboCode simulation platform. According to this strategy, a new model is proposed for the RoboCode platform, providing an environment for simulated robots that can be programmed to battle against other robots. Compared to Atari Games, RoboCode has a fairly wide set of actions and situations. Due to the challenges of training a CNN model for such a continuous action space problem, the inputs obtained from the simulation environment were generated dynamically, and the proposed model was trained by using these inputs. The trained model battled against the predefined rival robots of the environment (standard robots) by cumulatively benefiting from the experience of these robots. The comparison between the proposed model and standard robots of RoboCode Platform was statistically verified. Finally, the performance of the proposed model was compared with machine learning based-customized robots (community robots). Experimental results reveal that the proposed model is mostly superior to community robots. Therefore, the deep Q-learning-based model has proven to be successful in such a complex simulation environment. It should also be noted that this new model facilitates simulation performance in adaptive and partially cluttered environments.

Control and Simulation of a 6-DOF Biped Robot based on Twin Delayed Deep Deterministic Policy Gradient Algorithm

Objectives: To study an algorithm to control a bipedal robot to walk so that it has a gait close to that of a human. It is known that the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is a highly efficient algorithm with a few changes compared to the popular algorithm — the commonly used Deep Deterministic Policy Gradient (DDPG) in the continuous action space problem in Reinforcement Learning. Methods: Different from the usual sparse reward function model used, in this study, a reward model combined with a sparse reward function and dense reward function will be proposed. The application of the TD3 algorithm together with the proposed reward function model to control a bipedal robot model with 6 degrees of freedom will be presented. The training process is simulated in Gazebo/Robot Operating System (ROS) environment. Finding: The results show that, when choosing a reward model combined with a sparse reward function and a dense reward function suitable for the robot model, will help it learn faster and achieve better results. The biped robot can walk straight with an almost human-like gait. In the paper, the results from the TD3 algorithm combined with the proposed reward model are also compared with the results from other algorithms. Novelty: Applying the TD3 algorithm combined with the proposed reward model for the 6-DOF biped robot and simulating the robot’s gait in Gazebo/ROS environment, ROS is a middleware that can be used to control a robot in a real environment in the future. Keywords: TD3; biped robot; reinforcement learning; ROS; Gazebo

S-shaped versus V-shaped transfer functions for binary Manta ray foraging optimization in feature selection problem

Feature selection (FS) is considered as one of the core concepts in the areas of machine learning and data mining which immensely impacts the performance of classification model. Through FS, irrelevant or partially relevant features can be eliminated which in turn helps in enhancing the performance of the model. Over the years, researchers have applied different meta-heuristic optimization techniques for the purpose of FS as these overcome the limitations of traditional optimization approaches. Going by the trend, we introduce a new FS approach based on a recently proposed meta-heuristic algorithm called Manta ray foraging optimization (MRFO) which is developed following the food foraging nature of the Manta rays, one of the largest known marine creatures. As MRFO is apposite for continuous search space problems, we have adapted a binary version of MRFO to fit it into the problem of FS by applying eight different transfer functions belonging to two different families: S-shaped and V-shaped. We have evaluated the eight binary versions of MRFO on 18 standard UCI datasets. Of these, the best one is considered for comparison with 16 recently proposed meta-heuristic FS approaches. The results show that MRFO outperforms the state-of-the-art methods in terms of both classification accuracy and number of features selected. The source code of this work is available in https://github.com/Rangerix/MetaheuristicOptimization .

Energy-Efficient Mode Selection and Resource Allocation for D2D-Enabled Heterogeneous Networks: A Deep Reinforcement Learning Approach

Improving energy efficiency has shown increasing importance in designing future cellular system. In this work, we consider the issue of energy efficiency in D2D-enabled heterogeneous cellular networks. Specifically, communication mode selection and resource allocation are jointly considered with the aim to maximize the energy efficiency in the long term. And an Markov decision process (MDP) problem is formulated, where each user can switch between traditional cellular mode and D2D mode dynamically. We employ deep deterministic policy gradient (DDPG), a model-free deep reinforcement learning algorithm, to solve the MDP problem in continuous state and action space. The architecture of proposed method consists of one actor network and one critic network. The actor network uses deterministic policy gradient scheme to generate deterministic actions for agent directly, and the critic network employs value function based Q networks to evaluate the performance of the actor network. Simulation results show the convergence property of proposed algorithm and the effectiveness in improving the energy efficiency in a D2D-enabled heterogeneous network.

Orientational variable-length strip covering problem: A branch-and-price-based algorithm

In this article, we study a challenging optimization problem, namely the orientational variable-length strip covering problem. Given a large rectangular region and multiple orientational strips with variable lengths, the strips should be positioned and their lengths be determined such that their union can cover the large region with the minimal total area. This problem is motivated by the real-world problem in which multiple Earth observation satellites are used to image a large region in a cooperative pattern. It is a challenging nonlinear combinatorial optimization problem in continuous space. We propose a set-covering-problem-like integer programming formulation for the problem based on grid discretization and prove that the optimal solutions can be achieved when the strips take discrete values. As there is a large number of columns, we use a column generation method to solve the linear relaxation problem and an enumeration algorithm to solve the subproblem. To accelerate the solution, we propose a provable dominance rule to greatly reduce the solution space of the subproblem, enabling the application of an implicit enumeration (IE) algorithm. Then, we propose a cell-gathered approximate pricing method based on the definition of nested father–child grids. As a result, an efficient branch-and-price-based algorithm (BPBA) is developed. The numerical tests on random instances show that the dominance rule can reduce the subproblem’s solutions by almost 74% and the BPBA can run five times faster than Gurobi (commercial solving software) to find comparable solutions on average.

Fermionic neural-network states for ab-initio electronic structure

Neural-network quantum states have been successfully used to study a variety of lattice and continuous-space problems. Despite a great deal of general methodological developments, representing fermionic matter is however still early research activity. Here we present an extension of neural-network quantum states to model interacting fermionic problems. Borrowing techniques from quantum simulation, we directly map fermionic degrees of freedom to spin ones, and then use neural-network quantum states to perform electronic structure calculations. For several diatomic molecules in a minimal basis set, we benchmark our approach against widely used coupled cluster methods, as well as many-body variational states. On some test molecules, we systematically improve upon coupled cluster methods and Jastrow wave functions, reaching chemical accuracy or better. Finally, we discuss routes for future developments and improvements of the methods presented.

An enhanced Moth-flame optimization algorithm for permutation-based problems

Moth-flame optimizer (MFO) is one of the recently proposed metaheuristic optimization techniques which has been successfully used in wide range of applications. However, there are two issues with the MFO algorithm. First, as a stochastic technique, MFO may prematurely converge at some local minima during the search process. Second, the original MFO was developed for continuous search space problems and is not directly applicable to, e.g., permutation-based problems (PBP). In this paper, a novel perturbation strategy is introduced to the MFO algorithm to avoid probable local minima regions. This strategy works as follows: if the best solution obtained so far doesn’t improve for a given number of consecutive iterations, the current population of solutions is perturbed using some crossover mechanism as an attempt to explore new promising neighbourhoods in the search space. In addition, smallest position values mapping technique is employed in order for the proposed, termed CrossMFO (COMFO), algorithm to be applicable to PBP problems. It is noticed that, despite these modifications, the proposed COMFO has the same time complexity order as the original MFO. Extensive simulation experiments are conducted to compare the proposed COMFO to the MFO, other enhanced versions of MFO, and some metaheuristic optimizers in solving the well-known Travelling Salesman Problem (TSP). Empirical results show that the solutions obtained using MFO are improved by a factor of 24–47% on average for large TSP instances having more than 100 cities using COMFO and can even reach 38–58% using different settings. In addition, compared to other algorithms in the literature, the proposed algorithm provides, on average, better solutions. Hence, it can be considered a promising and efficient technique for this type of problems.

Genetic algorithm for obstacle location-allocation problems with customer priorities

In this paper we propose a metaheuristic approach to solve a customer priority based location-allocation problem in presence of obstacles and location-dependent supplier capacities. In many network optimization problems presence of obstacles prohibits feasibility of a regular network design. This includes a wide range of applications including disaster relief and pandemic disease containment problems in healthcare management. We focus on this application since fast and efficient allocation of suppliers to demand nodes is a critical process that impacts the results of the containment strategy. In this study, we propose an integrated mixed-integer program with location-based capacity decisions that considers customer priorities in the network design. We propose an efficient multi-stage genetic algorithm that solves the problem in continuous space. The computational findings show the best allocation strategies derived from proposed algorithms.