Continuous Action Spaces Research Articles

A PDDQNLP Algorithm for Energy Efficient Computation Offloading in UAV-Assisted MEC

Unmanned aerial vehicle (UAV)-assisted mobile edge computing (MEC) is a promising technology to provide computational services to ground terminal devices (TDs) in remote areas or for emergency incidents with its flexibility and mobility. This paper aims to maximize the UAV’s energy efficiency while considering the fairness of offloading. We formulate this optimization problem by jointly considering the UAV flight time, the UAV 3D trajectory, the TD binary offloading decisions, and the time allocated to TDs, which is a mixed-integer nonlinear programming problem. The problem is transformed into two sub-problems and we propose a PDDQNLP (parametrized dueling deep Q-network and linear programming) algorithm based on the combination of deep reinforcement learning (DRL) and linear programming (LP) to address them. For the first sub-problem, a DRL-based algorithm is used to optimize the TD offloading decisions, the UAV trajectory, and the UAV flight time. The action space is hybrid that contains discrete actions (e.g., binary offloading) and continuous actions (e.g., UAV flight time). Therefore, we parameterize the action space and propose the PDDQN algorithm, which combines the DDPG algorithm for handling the continuous action space and the Dueling DQN algorithm for handling the discrete action space. According to the solution obtained from the first sub-problem, the LP is used to adjust the time allocated to TDs in the second sub-problem. The numerical results show that the PDDQNLP algorithm outperforms its counterparts.

A Reinforcement Learning Approach for Continuum Robot Control

Rigid joint manipulators are limited in their movement and degrees of freedom (DOF), while continuum robots possess a continuous backbone that allows for free movement and multiple DOF. Continuum robots move by bending over a section, taking inspiration from biological manipulators such as tentacles and trunks. This paper presents an implementation of a forward kinematics and velocity kinematics model to describe the planar continuum robot, along with the application of reinforcement learning (RL) as a control algorithm. In this paper, we have adopted the planar constant curvature representation for the forward kinematic modeling. This choice was made due to its straightforward implementation and its potential to fill the literature gap in the field RL-based control for planar continuum robots. The intended control mechanism is achieved through the use of Deep Deterministic Policy Gradient (DDPG), a RL algorithm that is suited for learning controls in continuous action spaces. After simulating the algorithm, it was observed that the planar continuum robot can autonomously move from any initial point to any desired goal point within the task space of the robot. By analyzing the results, we wanted to recommend a future direction for research in the field of continuum robot control, specifically in the application of RL algorithms. One potential area of focus could be the integration of sensory feedback, such as vision or force sensing, to improve the robot’s ability to navigate complex environments. Additionally, exploring the use of different RL algorithms, such as Proximal Policy Optimization (PPO) or Trust Region Policy Optimization (TRPO), could lead to further advancements in the field. Overall, this paper demonstrates the potential for RL-based control of continuum robots and highlights the importance of continued research in this area.

Computing Offloading Based on TD3 Algorithm in Cache-Assisted Vehicular NOMA-MEC Networks.

In this paper, in order to reduce the energy consumption and time of data transmission, the non-orthogonal multiple access (NOMA) and mobile edge caching technologies are jointly considered in mobile edge computing (MEC) networks. As for the cache-assisted vehicular NOMA-MEC networks, a problem of minimizing the energy consumed by vehicles (mobile devices, MDs) is formulated under time and resource constraints, which jointly optimize the computing resource allocation, subchannel selection, device association, offloading and caching decisions. To solve the formulated problem, we develop an effective joint computation offloading and task-caching algorithm based on the twin-delayed deep deterministic policy gradient (TD3) algorithm. Such a TD3-based offloading (TD3O) algorithm includes a designed action transformation (AT) algorithm used for transforming continuous action space into a discrete one. In addition, to solve the formulated problem in a non-iterative manner, an effective heuristic algorithm (HA) is also designed. As for the designed algorithms, we provide some detailed analyses of computation complexity and convergence, and give some meaningful insights through simulation. Simulation results show that the TD3O algorithm could achieve lower local energy consumption than several benchmark algorithms, and HA could achieve lower consumption than the completely offloading algorithm and local execution algorithm.

An application of deep reinforcement learning and vendor-managed inventory in perishable supply chain management

This article delves into the challenging supply chain management domain, explicitly addressing the intricate issue of perishable inventory allocation within a two-echelon supply chain. The approach outlined here leverages deep reinforcement learning with a keen understanding of the inherent stochasticity arising from uncertain demands and variable supply conditions. The examined supply chain encompasses two retailers and a central distribution center operating under a vendor-managed inventory system. The primary goal of this research is to combat the prevalent problems of wastage and shortages frequently encountered at the retail level in such supply chains. The study employs the Advantage Actor-Critic (A2C) algorithm, tailored to handle the continuous action space inherent in inventory allocation. To rigorously evaluate this approach, empirical data from a real-world blood supply chain in Tabriz is used for numerical experiments. This practical case study involves a single distribution center and two hospitals. The outcomes of these experiments affirm the effectiveness of the A2C algorithm, showcasing its ability to address the complex inventory allocation problem successfully. Furthermore, the research highlights that the algorithm outperforms existing supply chain policies, underscoring the pivotal role of optimal allocation in enhancing efficiency and operational excellence in perishable supply chains.

Deep reinforcement learning based trajectory design and resource allocation for task-aware multi-UAV enabled MEC networks

Computing tasks in the air is an important form of mobile edge computing (MEC) to improve the quality of service and enhance network coverage. In this paper, we investigate a multi-UAV cooperative computing model and massive devices access scenario in a service area without infrastructure. There are various types of ground devices with different tasks. Moreover, we consider that the UAV executing tasks of devices need to cache the content that task required. Therefore, we propose a multi-UAV enabled MEC network based on task awareness where each UAV caches some programs to execute tasks offloaded from devices. To minimize completion time, a joint UAV trajectory design, access decision and resource allocation problem is formulated. To address this intractable mixed integer non-linear programming problem, a multi-agent trajectory design and resource allocation (MATR) algorithm is proposed, where the multi-agent deep deterministic policy gradient (MADDPG) is applied. Considering the complexity of high-dimensional continuous action space, we introduce the particle swarm optimization (PSO) algorithm to jointly optimize access decisions, and computation resource allocation to reduce action space. In addition, we discuss the impact of the size of UAV cache space and the location of ground devices on the completion time. Simulation results show that the MATR algorithm can significantly reduce the completion time compared to the baselines.

Personalized robotic control via constrained multi-objective reinforcement learning

Reinforcement learning is capable of providing state-of-art performance in end-to-end robotic control tasks. Nevertheless, many real-world control tasks necessitate the balancing of multiple conflicting objectives while simultaneously ensuring that the learned policies adhere to constraints. Additionally, individual users may typically prefer to explore the personalized and diversified robotic control modes via specific preferences. Therefore, this paper presents a novel constrained multi-objective reinforcement learning algorithm for personalized end-to-end robotic control with continuous actions, allowing a trained single model to approximate the Pareto optimal policies for any user-specified preferences. The proposed approach is formulated as a constrained multi-objective Markov decision process, incorporating a nonlinear constraint design to facilitate the agent in learning optimal policies that align with specified user preferences across the entire preference space. Meanwhile, a comprehensive index based on hypervolume and entropy is presented to measure the convergence, diversity and evenness of the learned control policies. The proposed scheme is evaluated on nine multi-objective end-to-end robotic control tasks with continuous action space, and its effectiveness is demonstrated in comparison with the competitive baselines, including classical and state-of-the-art algorithms.

Optimization of Robotic Arm Grasping through Fractional-Order Deep Deterministic Policy Gradient Algorithm

With the rapid development of robotics technology, robotic arm grasping has gained significant attention in the fields of automation and artificial intelligence. In this study, we propose a fractional-order deep deterministic policy gradient (DDPG) algorithm for optimizing robotic arm grasping tasks. Traditional machine learning algorithms face challenges in handling continuous action spaces, while the DDPG algorithm effectively addresses this issue. In this research, we first review the background and challenges of robotic arm grasping and provide an overview of the application of traditional reinforcement learning algorithms in grasping tasks. Subsequently, we introduce the principles and fundamental ideas of the DDPG algorithm in detail, discussing its potential for optimizing robotic arm grasping. To further enhance the performance of robotic arm grasping, we propose an improved approach based on fractional-order control. Fractional-order control exhibits unique advantages in environmental dynamics modeling and grasp posture optimization, enhancing the robustness and adaptability of robotic arm grasping. Through a series of experiments, we validate the effectiveness and superiority of the fractional-order DDPG algorithm in robotic arm grasping tasks. Our algorithm achieves significant improvements in grasping success rate and stability compared to traditional methods. The experimental results demonstrate that the fractional-order DDPG algorithm is better equipped to handle control challenges in continuous action spaces and optimize the performance of robotic arm grasping tasks.

Reinforcement learning algorithms: A brief survey

Reinforcement Learning (RL) is a machine learning (ML) technique to learn sequential decision-making in complex problems. RL is inspired by trial-and-error based human/animal learning. It can learn an optimal policy autonomously with knowledge obtained by continuous interaction with a stochastic dynamical environment. Problems considered virtually impossible to solve, such as learning to play video games just from pixel information, are now successfully solved using deep reinforcement learning. Without human intervention, RL agents can surpass human performance in challenging tasks. This review gives a broad overview of RL, covering its fundamental principles, essential methods, and illustrative applications. The authors aim to develop an initial reference point for researchers commencing their research work in RL. In this review, the authors cover some fundamental model-free RL algorithms and pathbreaking function approximation-based deep RL (DRL) algorithms for complex uncertain tasks with continuous action and state spaces, making RL useful in various interdisciplinary fields. This article also provides a brief review of model-based and multi-agent RL approaches. Finally, some promising research directions for RL are briefly presented.

Reentry trajectory design of a hypersonic vehicle based on reinforcement learning

In this research, we investigate control of a hypersonic vehicle (HV) following its reentry into the Earth’s atmosphere, using deep reinforcement learning (DRL) in a continuous space. Here, we incorporate the basic kinematic and force equations of motion for a vehicle in an atmospheric flight to formulate the reentry trajectory satisfying the boundary constraints and multiple mission related process constraints. The aerodynamic model of the vehicle emulates the properties of a common aero vehicle (CAV-H), while the atmospheric model of the Earth represents a standard model based on US Standard Atmosphere 1976, with significant simplification to the planetary model. In an unpowered flight, we then control the vehicle’s trajectory by perturbing its angle of attack and bank angle to achieve the desired objective, where the control problem is based on different actor-critic frameworks that utilize neural networks (NNs) as function approximators to select and evaluate the control actions in continuous state and action spaces. First, we train the model following each of the methods, that include on-policy proximal policy approximation (PPO) and off-policy twin delayed deterministic policy gradient (TD3). From the trajectory generated, we select a nominal trajectory for each algorithm that satisfies our mission requirements based on the reward model.

Latency-aware task offloading in UAV-assisted edge computing: Leveraging deep reinforcement learning

Due to the exponential expansion of the Internet of Things (IoT) technology, the increasing amount of data generated by numerous IoT devices has become a significant challenge for communication and computing networks. On the other hand, the concept of Mobile Edge Computing (MEC) presents a viable solution as it brings computing, storage, and networking capabilities in close proximity to the user, enabling the hosting of applications that require significant computation power and low latency right at the network’s edge. In addition, the maneuverability, cost-effectiveness, and ease of deployment make Unmanned Aerial Vehicles (UAVs) highly versatile wireless platforms. Capitalizing on the benefits of MEC and UAVs, this paper proposes a UAV-assisted MEC system that offloads services to reduce the computation burden in IoT. The objective is to maximize the task completion rate, considering factors such as user transmit power, task offloading rate, and UAV trajectory variables. To tackle this optimization problem, this paper devises a method based on the Deep Deterministic Policy Gradient (DDPG), an algorithm for continuous action spaces in deep reinforcement learning. The numerical simulations show the effectiveness of the proposed approach in comparison to other benchmarks, i.e., deep Q-network (DQN) and full offloading methods. In a simulation, the proposed DDPG method can achieve a fully optimized system with a 100% task success rate, while the DQN-based method achieves a lower task success rate of 95.93% and the full offloading method only obtains a task success rate of 82.73%.

Leveraging Machine Learning for Generating and Utilizing Motion Primitives in Adversarial Environments

Motion primitives enable fast planning for complex and dynamic environments. Adversarial environments pose a particularly challenging and unpredictable scenario. Motion-primitive-based planners have the potential to provide benefit in these types of environments. The key challenge is to design a library of maneuvers that effectively capture the necessary capabilities of the vehicle. This work presents a primitive-based game tree search to solve adversarial games in continuous state and action spaces and applies a reinforcement learning framework to autonomously generate effective primitives for the given task. The results demonstrate the ability of the learning framework to produce maneuvers necessary for competing against adversaries. Furthermore, we propose a method for learning a model to estimate the state-dependent value of each motion primitives and demonstrate how to incorporate this model to increase planning performance under time constraints. Additionally, we compare our primitive-based algorithm against forward simulated methods from existing literature and highlight the benefits of motion primitives.

Self-Organizing Memory Based on Adaptive Resonance Theory for Vision and Language Navigation

Vision and Language Navigation (VLN) is a task in which an agent needs to understand natural language instructions to reach the target location in a real-scene environment. To improve the model ability of long-horizon planning, emerging research focuses on extending the models with different types of memory structures, mainly including topological maps or a hidden state vector. However, the fixed-length hidden state vector is often insufficient to capture long-term temporal context. In comparison, topological maps have been shown to be beneficial for many robotic navigation tasks. Therefore, we focus on building a feasible and effective topological map representation and using it to improve the navigation performance and the generalization across seen and unseen environments. This paper presents a S elf-organizing Memory based on Adaptive Resonance Theory (SMART) module for incremental topological mapping and a framework for utilizing the SMART module to guide navigation. Based on fusion adaptive resonance theory networks, the SMART module can extract salient scenes from historical observations and build a topological map of the environmental layout. It provides a compact spatial representation and supports the discovery of novel shortcuts through inferences while being explainable in terms of cognitive science. Furthermore, given a language instruction and on top of the topological map, we propose a vision–language alignment framework for navigational decision-making. Notably, the framework utilizes three off-the-shelf pre-trained models to perform landmark extraction, node–landmark matching, and low-level controlling, without any fine-tuning on human-annotated datasets. We validate our approach using the Habitat simulator on VLN-CE tasks, which provides a photo-realistic environment for the embodied agent in continuous action space. The experimental results demonstrate that our approach achieves comparable performance to the supervised baseline.

Secrecy Rate Maximization in THz-Aided Heterogeneous Networks: A Deep Reinforcement Learning Approach

Densely deployed femtocells in heterogeneous networks (HetNets) improve the quality-of-service (QoS) and quality-of-experience (QoE) for both next-generation networks and end users, respectively. However, users near the femtocell edge, i.e., femto edge users (FEUs), which are far away from the Femto base station (FBS) may receive a low signal-to-interference-plus-noise ratio (SINR). Moreover, due to the distributed and dynamic nature of HetNets, femtocells may not resist various eavesdropping attacks from different attackers. Thus, it is essential to ensure the wireless channel's secrecy through which FEUs communicate with FBS since they may face eavesdropping attacks and may have low SINR and achievable data rates. Aiming to enhance the secrecy rate of femtocells, in this article, we formulate a joint optimization problem of beamforming vectors, artificial noise (AN), and power control for terahertz (THz)-enabled femtocells. Addressing this non-convex optimization problem is challenging due to the system's complexity and dynamic nature of FEUs. Thus, we translate the optimization problem into a multi-agent reinforcement learning (MARL) problem using the Markov decision process (MDP). Then, to solve the MDP with continuous action space, two deep reinforcement learning (DRL) based schemes, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Deep Deterministic Policy Gradient-based Secrecy Maximization (DDPG-SM)</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Asynchronous-Advantage-Actor-Critic based Secrecy Maximization (A3C-SM)</i> are proposed to maximize the secrecy rate of femtocells. The secrecy rate performance of the proposed schemes is compared and analyzed with Advantage-Actor-Critic (A2C) and state-of-the-art Perfect Secrecy Rate Maximization (PSRM) schemes. Simulation results show that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> schemes achieve 37.73% and 22.64% better average secrecy rate in comparison to the PSRM scheme. Also, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> scheme outperforms all other <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> , A2C, and PSRM schemes in terms of average secrecy rate, received SINR, and energy-efficiency of femtocells.

A Continuous Action Space Tree search for INverse desiGn (CASTING) framework for materials discovery

Material properties share an intrinsic relationship with their structural attributes, making inverse design approaches crucial for discovering new materials with desired functionalities. Reinforcement Learning (RL) approaches are emerging as powerful inverse design tools, often functioning in discrete action spaces. This constrains their application in materials design problems, which involve continuous search spaces. Here, we introduce an RL-based framework CASTING (Continuous Action Space Tree Search for inverse design), that employs a decision tree-based Monte Carlo Tree Search (MCTS) algorithm with continuous space adaptation through modified policies and sampling. Using representative examples like Silver (Ag) for metals, Carbon (C) for covalent systems, and multicomponent systems such as graphane, boron nitride, and complex correlated oxides, we showcase its accuracy, convergence speed, and scalability in materials discovery and design. Furthermore, with the inverse design of super-hard Carbon phases, we demonstrate CASTING’s utility in discovering metastable phases tailored to user-defined target properties and preferences.

Deep reinforcement learning based energymanagement strategy considering running costs and energy source aging for fuel cell hybrid electric vehicle

The main contribution of this study is to integrate energy source aging and running costs into the deep reinforcement learning (DRL) based EMS of fuel cell hybrid electric vehicles (FCHEV). For the FCHEV, a multi-objective energy management strategy (EMS) based on twin delayed deep deterministic policy gradient (TD3) is proposed, which aims to simultaneously reduce energy source degradation and lower running costs. To achieve this, the paper innovatively designs the reward function and it's comparative approach. Additionally, it verifies the superiority of the proposed EMS over other EMS based on continuous action space algorithm, including previous action guided deep deterministic policy gradient (PA-DDPG) and soft actor-critic (SAC). Lastly, the agent's action output is changed from fuel cell (FC) current to FC power ratio, and a comparative analysis on results generated by different action outputs is conducted. Simulation results show that the proposed EMS can reduce the running costs while extending the lifespan of battery and FC efficiently. This work holds significant practical significance in the energy distribution of automobiles.

A deep Q-learning based algorithmic trading system for commodity futures markets

Nowadays, investors seek more sophisticated decision-making tools that maximize their profit from investing in the financial markets by suitably determining the optimal position, trading time, price, and volume. This paper proposes a novel intraday algorithmic trading system for volatile commodity futures markets based on a Deep Q-network (DQN) algorithm and its robust double-version (DDQN). The higher volatility, leverage property, and more liquidity in futures contracts give investors more opportunity to take advantage of speculative behaviors with a relatively small amount of capital; however, the volatility brings more difficulties in the learning phase. As an essential prerequisite to training and evaluating any trading algorithm in the futures market, we develop a simulator to replicate a real futures exchange market environment that executes recommended trading signals by handling the clearing and margin management and the pre-order checking mechanisms. Moreover, this study provides a new definition of the continuous state and action spaces that match the futures market's characteristics. To address the curse of dimensionality, we utilize a multi-agent architecture equipped with the Gated Recurrent Unit (GRU) networks to approximate the Q-values functions. The experimental results demonstrate that implementing the proposed trading algorithms (especially the DDQN) into the actual intraday data of gold coin futures contracts significantly outperforms the benchmarks in terms of return, risk, and risk-adjusted return.

Intercept Guidance of Maneuvering Targets with Deep Reinforcement Learning

In this paper, a novel guidance law based on a reinforcement learning (RL) algorithm is presented to deal with the maneuvering target interception problem using a deep deterministic policy gradient descent neural network. We take the missile’s line-of-sight (LOS) rate as the observation of the RL algorithm and propose a novel reward function, which is constructed with the miss distance and LOS rate to train the neural network off-line. In the guidance process, the trained neural network has the capacity of mapping the missile’s LOS rate to the normal acceleration of the missile directly, so as to generate guidance commands in real time. Under the actor-critic (AC) framework, we adopt the twin-delayed deep deterministic policy gradient (TD3) algorithm by taking the minimum value between a pair of critics to reduce overestimation. Simulation results show that the proposed TD3-based RL guidance law outperforms the current state of the RL guidance law, has better performance to cope with continuous action and state space, and also has a faster convergence speed and higher reward. Furthermore, the proposed RL guidance law has better accuracy and robustness when intercepting a maneuvering target, and the LOS rate is converged.

A New Fourier Q Operator Network Based Reinforcement Learning Method for Continuous Action Space Decision-making in Manufacturing

The problems of continuous action space decision-making are widespread in industrial manufacturing. However, when dealing with these problems, existing reinforcement learning (RL) methods relies on a large number of training samples, which is always unacceptable given the limited availability or expensive nature of data, such as low-volume manufacturing. This paper proposes a new Fourier Q operator network (FQON) based RL method. The input of FQON is the expected state function and its output the Q-value function, and both functions take the action in RL as independent variables. The infinite-dimensional mapping between the function domains is established by a set of parameters that can be used with different discretization, which fixes the mapping complexity regardless of the action space resolution. By taking the advantages of the fast calculation using on Fourier kernel operator, the mapping complexity is highly reduced, and it enables that FQON can realize the decision-making in continuous action space using a small amount of training samples. Taking machining deformation control of an aero-engine casing as a case study, experimental results showed that FQON based RL method can control the deformation well with limited training samples.

Incentive-based demand response under incomplete information based on the deep deterministic policy gradient

Incentive-based demand response (IBDR), as an important measure to encourage the users to participate in the demand-side management, is commonly modeled as the Stackelberg game with the complete information. However, it is difficult to acquire the users' complete information due to privacy protections. In this paper, a Markov decision process (MDP) game model is proposed to address IBDR under the incomplete information, which is based on a deep deterministic policy gradient algorithm (DDPG). Considering differences on the users' load, the K-means method is used to classify different users according to the daily load rate and the peak-to-valley difference, such that different types of user load will have different incentive prices to participate in demand responses. The proposed DDPG algorithm can improve the calculation efficiency of the Nash equilibrium solution of the IBDR under the incomplete information, as it can deal with the multi-dimensional continuous state and action spaces. Simulation results show that the proposed approach can achieve the Nash equilibrium under the incomplete information and has the higher calculation accuracy and the lower calculation time in comparison to the Q learning method.

A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat

Unmanned combat air vehicle (UCAV) combat is a challenging scenario with high-dimensional continuous state and action space and highly nonlinear dynamics. In this article, we propose a general hierarchical framework to resolve the within-vision-range (WVR) air-to-air combat problem under six dimensions of degree (6-DOF) dynamics. The core idea is to divide the whole decision-making process into two loops and use reinforcement learning (RL) to solve them separately. The outer loop uses a combat policy to decide the macro command according to the current combat situation. Then the inner loop uses a control policy to answer the macro command by calculating the actual input signals for the aircraft. We design the Markov decision-making process for the control policy and the Markov game between two aircraft. We present a two-stage training mechanism. For the control policy, we design an effective reward function to accurately track various macro behaviors. For the combat policy, we present a fictitious self-play mechanism to improve the combat performance by combating against the historical combat policies. Experiment results show that the control policy can achieve better tracking performance than conventional methods. The fictitious self-play mechanism can learn competitive combat policy, which can achieve high winning rates against conventional methods.