Continuous Action Spaces Research Articles

Multi-IRSs enhanced UAV secure semantic communication system

This paper delves into a unmanned aerial vehicle (UAV) semantic communication system featuring multiple intelligent reflecting surfaces (IRSs). The UAV transmits semantic symbols to users in the presence of eavesdropping, and thus has inevitable security problems. In order to measure secure communication performance in semantic communication, the security semantic transmission rate (SSR) based on the semantic similarity is employed. An optimization problem is formulated to enhance the average SSR by optimizing UAV trajectory, IRS reflecting coefficients, number of semantic symbols, and transmission power, while satisfying position, speed, anti-collision, and power constraints. Due to the highly dynamic nature of the environment and the difficulties in solving complex modeling problems, a novel approach based on hybrid soft actor–critic (H-SAC) deep reinforcement learning (DRL) is proposed to address this non-convex optimization problem. The algorithm addresses the challenge of dynamic environments in UAV scenarios by seamlessly accommodating both discrete and continuous action spaces such as the trajectory and the IRS reflecting coefficient within the network. This enables effective processing of actions taken by intelligent agents, ensuring adaptability to the diverse requirements of the environment without the need for separate handling mechanisms. Simulation results show that the proposed intelligent algorithm exhibits excellent convergence and achieves a well-balanced pairing of appropriate IRSs and selection of flight paths, significantly improving the average SSR of the system. Specifically, our proposed scheme enhances the secure semantic transfer rate by more than fifty percent compared to when the number of semantic symbols is fixed at two.

Modeling coupled driving behavior during lane change: A multi-agent Transformer reinforcement learning approach

In a lane change (LC) scenario, the lane change vehicle interacts with surrounding vehicles. The interactions not only affect their driving behaviors but also influence the traffic flow. This study aims to model the coupled behavior of the lane changer and the follower in the target lane during LC. Large-scale real-world connected vehicle (CV) data from the Safety Pilot Model Deployment (SPMD) program are used to extract LCs and study vehicle interactions. A multi-agent Transformer-based deep deterministic policy gradient (MA-TDDPG) method is proposed to model the coupled behaviors during LC. The multi-agent framework can handle the multiple agents’ behaviors with interactions, and the Transformer can process the observation-action memory accurately and efficiently. The MA-TDDPG algorithm can learn the sequential decision-making process over continuous action space during LC with the accommodation of the multi-vehicle interaction and the driver’s memory effect. Compared to traditional supervised learning and reinforcement learning methods, it demonstrates superior performance in imitating the longitudinal and lateral actions of the lane changer and the follower. The findings of this study provide insights into the development of microscopic simulations by producing realistic LC behaviors, and assistance/automation LC systems by generating LC motions and responses conforming to human driving habits. The model also creates an interactive simulation environment and lays the foundation for optimizing driving strategies.

Entropy Enhanced Multiagent Coordination Based on Hierarchical Graph Learning for Continuous Action Space

Entropy Enhanced Multiagent Coordination Based on Hierarchical Graph Learning for Continuous Action Space

Path planning of mobile robot based on improved TD3 algorithm in dynamic environment

This paper proposes an improved TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm to address the flaws of low success rate and slow training speed, when using the original TD3 algorithm in mobile robot path planning in dynamic environment. Firstly, prioritized experience replay and transfer learning are introduced to enhance the learning efficiency, where the probability of beneficial experiences being sampled in the experience pool is increased, and the pre-trained model is applied in an obstacle-free environment as the initial model for training in a dynamic environment. Secondly, dynamic delay update strategy is devised and OU noise is added to improve the success rate of path planning, where the probability of missing high-quality value estimate is reduced through changing the delay update interval dynamically, and the correlated exploration of the mobile robot inertial navigation system in the dynamic environment is temporally improved. The algorithm is tested by simulation where the Turtlebot3 robot model as a training object, the ROS melodic operating system and Gazebo simulation software as an experimental environment. Meanwhile, the result shows that the improved TD3 algorithm has a 16.6 % increase in success rate and a 23.5 % reduction in algorithm training time. A generalization experiment was designed finally, and it indicates that superior generation performance has been acquired in mobile robot path planning with continuous action spaces through the improved TD3 algorithm.

Application of Hybrid Deep Reinforcement Learning for Managing Connected Cars at Pedestrian Crossings: Challenges and Research Directions

The autonomous vehicle is an innovative field for the application of machine learning algorithms. Controlling an agent designed to drive safely in traffic is very complex as human behavior is difficult to predict. An individual’s actions depend on a large number of factors that cannot be acquired directly by visualization. The size of the vehicle, its vulnerability, its perception of the environment and weather conditions, among others, are all parameters that profoundly modify the actions that the optimized model should take. The agent must therefore have a great capacity for adaptation and anticipation in order to drive while ensuring the safety of users, especially pedestrians, who remain the most vulnerable users on the road. Deep reinforcement learning (DRL), a sub-field that is supported by the community for its real-time learning capability and the long-term temporal aspect of its objectives looks promising for AV control. In a previous article, we were able to show the strong capabilities of a DRL model with a continuous action space to manage the speed of a vehicle when approaching a pedestrian crossing. One of the points that remains to be addressed is the notion of discrete decision-making intrinsically linked to speed control. In this paper, we will present the problems of AV control during a pedestrian crossing, starting with a modelization and a DRL model with hybrid action space adapted to the scalability of a vehicle-to-pedestrian (V2P) encounter. We will also present the difficulties raised by the scalability and the curriculum-based method.

Actor-Critic Reinforcement Learning Algorithms for Mean Field Games in Continuous Time, State and Action Spaces

Actor-Critic Reinforcement Learning Algorithms for Mean Field Games in Continuous Time, State and Action Spaces

Portfolio management based on a reinforcement learning framework

AbstractPortfolio management is crucial for investors. We propose a dynamic portfolio management framework based on reinforcement learning using the proximal policy optimization algorithm. The two‐part framework includes a feature extraction network and a full connected network. First, the majority of the previous research on portfolio management based on reinforcement learning has been dedicated to discrete action spaces. We propose a potential solution to the problem of a continuous action space with a constraint (i.e., the sum of the portfolio weights is equal to 1). Second, we explore different feature extraction networks (i.e., convolutional neural network [CNN], long short‐term memory [LSTM] network, and convolutional LSTM network) combined with our system, and we conduct extensive experiments on the six kinds of assets, including 16 features. The empirical results show that the CNN performs best in the test set. Last, we discuss the effect of the trading frequency on our trading system and find that the monthly trading frequency has a higher Sharpe ratio in the test set than other trading frequencies.

Beyond Static Obstacles: Integrating Kalman Filter with Reinforcement Learning for Drone Navigation

Autonomous drones offer immense potential in dynamic environments, but their navigation systems often struggle with moving obstacles. This paper presents a novel approach for drone trajectory planning in such scenarios, combining the Interactive Multiple Model (IMM) Kalman filter with Proximal Policy Optimization (PPO) reinforcement learning (RL). The IMM Kalman filter addresses state estimation challenges by modeling the potential motion patterns of moving objects. This enables accurate prediction of future object positions, even in uncertain environments. The PPO reinforcement learning algorithm then leverages these predictions to optimize the drone’s real-time trajectory. Additionally, the capability of PPO to work with continuous action spaces makes it ideal for the smooth control adjustments required for safe navigation. Our simulation results demonstrate the effectiveness of this combined approach. The drone successfully navigates complex dynamic environments, achieving collision avoidance and goal-oriented behavior. This work highlights the potential of integrating advanced state estimation and reinforcement learning techniques to enhance autonomous drone capabilities in unpredictable settings.

Control of Qubit Dynamics Using Reinforcement Learning

The progress in machine learning during the last decade has had a considerable impact on many areas of science and technology, including quantum technology. This work explores the application of reinforcement learning (RL) methods to the quantum control problem of state transfer in a single qubit. The goal is to create an RL agent that learns an optimal policy and thus discovers optimal pulses to control the qubit. The most crucial step is to mathematically formulate the problem of interest as a Markov decision process (MDP). This enables the use of RL algorithms to solve the quantum control problem. Deep learning and the use of deep neural networks provide the freedom to employ continuous action and state spaces, offering the expressivity and generalization of the process. This flexibility helps to formulate the quantum state transfer problem as an MDP in several different ways. All the developed methodologies are applied to the fundamental problem of population inversion in a qubit. In most cases, the derived optimal pulses achieve fidelity equal to or higher than 0.9999, as required by quantum computing applications. The present methods can be easily extended to quantum systems with more energy levels and may be used for the efficient control of collections of qubits and to counteract the effect of noise, which are important topics for quantum sensing applications.

Local 2-D Path Planning of Unmanned Underwater Vehicles in Continuous Action Space Based on the Twin-Delayed Deep Deterministic Policy Gradient

Local 2-D Path Planning of Unmanned Underwater Vehicles in Continuous Action Space Based on the Twin-Delayed Deep Deterministic Policy Gradient

An online hyper‐volume action bounding approach for accelerating the process of deep reinforcement learning from multiple controllers

AbstractThis paper fuses ideas from reinforcement learning (RL), Learning from Demonstration (LfD), and Ensemble Learning into a single paradigm. Knowledge from a mixture of control algorithms (experts) are used to constrain the action space of the agent, enabling faster RL refining of a control policy, by avoiding unnecessary explorative actions. Domain‐specific knowledge of each expert is exploited. However, the resulting policy is robust against errors of individual experts, since it is refined by a RL reward function without copying any particular demonstration. Our method has the potential to supplement existing RLfD methods when multiple algorithmic approaches are available to function as experts, specifically in tasks involving continuous action spaces. We illustrate our method in the context of a visual servoing (VS) task, in which a 7‐DoF robot arm is controlled to maintain a desired pose relative to a target object. We explore four methods for bounding the actions of the RL agent during training. These methods include using a hypercube and convex hull with modified loss functions, ignoring actions outside the convex hull, and projecting actions onto the convex hull. We compare the training progress of each method using expert demonstrators, employing one expert demonstrator with the DAgger algorithm, and without using any demonstrators. Our experiments show that using the convex hull with a modified loss function not only accelerates learning but also provides the most optimal solution compared with other approaches. Furthermore, we demonstrate faster VS error convergence while maintaining higher manipulability of the arm, compared with classical image‐based VS, position‐based VS, and hybrid‐decoupled VS.

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Advancements in Reinforcement Learning

Reinforcement Learning (RL) stands at the forefront of machine learning, offering a paradigm where agents learn to navigate complex environments through trial and error interactions. This paper provides a comprehensive overview of RL concepts, algorithms, challenges, and future directions. Key concepts such as agent-environment interaction, Markov Decision Processes (MDPs), value-based and policy-based methods are elucidated. Challenges including the exploration-exploitation trade-off, sample efficiency, and safety concerns are discussed, alongside potential breakthroughs in deep RL, handling continuous action spaces, and dealing with partial observability. The integration of RL with emerging technologies like IoT and its implications for real-world applications such as robotics, autonomous vehicles, and healthcare are explored. Finally, future trends and directions in RL research, including AI engineering and automated feature engineering, are outlined, highlighting the potential for transformative advancements and their impact on various domains. This paper serves as a comprehensive resource for researchers, practitioners, and enthusiasts interested in the evolving landscape of Reinforcement Learning

Mechanism design for public projects via three machine learning based approaches

We study mechanism design for nonexcludable and excludable binary public project problems. Our aim is to maximize the expected number of consumers and the expected agents’ welfare. We first show that for the nonexcludable public project model, there is no need for machine learning based mechanism design. We identify a sufficient condition on the prior distribution for the existing conservative equal costs mechanism to be the optimal strategy-proof and individually rational mechanism. For general distributions, we propose a dynamic program that solves for the optimal mechanism. For the excludable public project model, we identify a similar sufficient condition for the existing serial cost sharing mechanism to be optimal for 2 and 3 agents. We derive a numerical upper bound and use it to show that for several common distributions, the serial cost sharing mechanism is close to optimality. The serial cost sharing mechanism is not optimal in general. We propose three machine learning based approaches for designing better performing mechanisms. We focus on the family of largest unanimous mechanisms, which characterizes all strategy-proof and individually rational mechanisms for the excludable public project model. A largest unanimous mechanism describes an iterative mechanism, which is defined by an exponential number of mechanism parameters. Our first approach describes the largest unanimous mechanism family using a neural network and training is carried out by minimizing a cost function that combines the mechanism design objective and the constraint violation penalty. We interpret the largest unanimous mechanisms as price-oriented rationing-free (PORF) mechanisms, which enables us to move the mechanisms’ iterative decision making off the neural network, to a separate simulation process, therefore avoiding the vanishing gradient problem. We also feed the prior distribution’s analytical form into the cost function to achieve high-quality gradients for efficient training. Our second approach treats the mechanism design task as a Markov Decision Process with an exponential number of states. During the Markov decision process, the non-consumers are gradually removed from the system. We train multiple neural networks, each for a different number of remaining agents, to learn the optimal value function on the states. Training is carried out by supervised learning toward a set of manually prepared base cases and the Bellman equation. Our third approach is based on reinforcement learning for a Partially Observable Markov Decision Process. Each RL episode randomly draws a type profile, which is hidden from the RL agent (mechanism designer). The RL agent only observes which cost share offers have been accepted under the largest unanimous mechanism under discussion. We use a continuous action space reinforcement learning approach to adjust the offer policy (i.e., adjust mechanism parameters). Lastly, our first two approaches use “supervision to manual mechanisms” as a systematic way for network initialization, which is potentially valuable for machine learning based mechanism design in general.

Real-time Gait Event Tracker (RGET): An Innovative Approach to Digital Human Modeling through Integration of Inertial Measurement Unit and Dynamic Feature Extraction

The advancement of digital human modeling technology underscores the importance of accurately representing human characteristics and behaviors using motion capture technologies. This paper introduces the Real-time Gait Event Tracker (RGET), an advanced method aimed at precisely detecting gait events via an Inertial Measurement Unit (IMU), thereby supporting high-precision digital human modeling. RGET integrates planning techniques into continuous states and action spaces, utilizing first-order difference functions and sliding window techniques to effectively identify four key gait events: Heel Strike (HS), Toe Off (TO), Walking Start (WS), and Walking Pause (WP). The system efficiently manages IMU gait signal data through real-time queuing techniques and extracts dynamic features within time frames using positive/negative windowed first-order difference integrations. By adopting weighted sleep time methods and adaptive threshold decision rules, RGET accurately extracts gait event features from sequences. Not only does RGET support precise gait segmentation with an F1 score of 95.9%, but it also provides an innovative method for digital human modeling and its applications.

Tugboat Scheduling Method Based on the NRPER-DDPG Algorithm: An Integrated DDPG Algorithm with Prioritized Experience Replay and Noise Reduction

The scheduling of harbor tugboats is a crucial task in port operations, aiming to optimize resource allocation and reduce operational costs, including fuel consumption of tugboats and the time cost of vessels waiting for operations. Due to the complexity of the port environment, traditional scheduling methods, often based on experience and practice, lack scientific and systematic decision support, making it difficult to cope with real-time changes in vessel dynamics and environmental factors. This often leads to scheduling delays and resource waste. To address this issue, this study proposes a mathematical model based on fuzzy programming, accounting for the uncertainty of the arrival time of target vessels. Additionally, we introduce the NRPER-DDPG algorithm (DDPG Algorithm with Prioritized Experience Replay and Noise Reduction), which combines a prioritized replay mechanism with a decaying noise strategy based on the DDPG algorithm. This approach optimizes the time for tugboats to reach the task location as a continuous action space, aiming to minimize the total system cost and improve scheduling efficiency. To verify the effectiveness of the mathematical model and algorithm, this study conducted experimental validation. Firstly, the optimal algorithm hyperparameter combinations were adjusted through random examples to ensure the stability and reliability of the algorithm. Subsequently, large-scale examples and actual port cases were used to further verify the performance advantages of the algorithm in practical applications. Experimental results demonstrate that the proposed mathematical model and algorithm significantly reduce system costs and improve scheduling efficiency, providing new insights and methods for the sustainable development of port operations.

AoI minimization of ambient backscatter-assisted EH-CRN with cooperative spectrum sensing

To evaluate the timeliness and freshness of information from energy-constrained devices, we minimize the weighted average age of information (AoI) of the ambient backscatter-assisted energy harvesting cognitive radio network (AB-EH-CRN), where secondary transmitters (STs) follow the non-orthogonal multiple access (NOMA) strategy to transmit status updates. The secondary receiver (SR) schedules the selected actions for STs to perform spectrum access, and receives status updates from them. We formulate the AoI minimization problem where the action selection and transmission power of STs are considered as variables. To exploit spectrum resources in the AB-EH-CRN with unknown signal-to-noise ratios (SNRs) at the STs, we propose the deep neural network (DNN)-based cooperative spectrum sensing (CSS), and adopt an energy threshold approach for STs to determine the energy allocation between spectrum sensing (SS) and packet transmission. Moreover, we address the AoI minimization problem by the proposed hybrid action and energy penalty deep deterministic policy gradient (HAEP-DDPG) algorithm, which adapts to the hybrid continuous and discrete action spaces. Simulation results validate the advantage of the proposed HAEP-DDPG algorithm compared with comparison schemes in terms of AoI.

Analyzing Queueing Problems via Bandits With Linear Reward & Nonlinear Workload Fairness

Queueing models serve as important building blocks in many networking applications such as task scheduling in mobile edge computing nodes, traffic scheduling in networks, congestion control in Internet, etc. However, queueing theory often needs to make strong assumptions about the arrival process or service rewards at each queue. In addition, fairness in serving workload among all queues is of great importance in many applications. In this paper, we address how to optimize resource allocation among multiple queues with a fairness guarantee and without any a priori knowledge of these queues' parameters. To characterize queues with unknown parameters and the fairness requirement, we formulate an online learning model with a varying and continuous action space, as well as a nonlinear utility objective. We design an online learning algorithm to tackle the problem. We prove that our algorithm has a regret upper bound of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(\sqrt{T}\log T)$</tex-math></inline-formula> and our model has a regret lower bound of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Omega (\sqrt{T})$</tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$T$</tex-math></inline-formula> stands for the number of decision rounds. The asymptotic closeness of upper and lower bounds guarantees their near tightness and our algorithm's near optimality. We discuss our model's real-world applications in mobile edge computing, wireless networks, and crowdsourcing, and conduct simulations to validate our algorithm's effectiveness.

Probabilistic Offline Policy Ranking with Approximate Bayesian Computation

In practice, it is essential to compare and rank candidate policies offline before real-world deployment for safety and reliability. Prior work seeks to solve this offline policy ranking (OPR) problem through value-based methods, such as Off-policy evaluation (OPE). However, they fail to analyze special case performance (e.g., worst or best cases), due to the lack of holistic characterization of policies’ performance. It is even more difficult to estimate precise policy values when the reward is not fully accessible under sparse settings. In this paper, we present Probabilistic Offline Policy Ranking (POPR), a framework to address OPR problems by leveraging expert data to characterize the probability of a candidate policy behaving like experts, and approximating its entire performance posterior distribution to help with ranking. POPR does not rely on value estimation, and the derived performance posterior can be used to distinguish candidates in worst-, best-, and average-cases. To estimate the posterior, we propose POPR-EABC, an Energy-based Approximate Bayesian Computation (ABC) method conducting likelihood-free inference. POPR-EABC reduces the heuristic nature of ABC by a smooth energy function, and improves the sampling efficiency by a pseudo-likelihood. We empirically demonstrate that POPR-EABC is adequate for evaluating policies in both discrete and continuous action spaces across various experiment environments, and facilitates probabilistic comparisons of candidate policies before deployment.

Risk-Aware Continuous Control with Neural Contextual Bandits

Recent advances in learning techniques have garnered attention for their applicability to a diverse range of real-world sequential decision-making problems. Yet, many practical applications have critical constraints for operation in real environments. Most learning solutions often neglect the risk of failing to meet these constraints, hindering their implementation in real-world contexts. In this paper, we propose a risk-aware decision-making framework for contextual bandit problems, accommodating constraints and continuous action spaces. Our approach employs an actor multi-critic architecture, with each critic characterizing the distribution of performance and constraint metrics. Our framework is designed to cater to various risk levels, effectively balancing constraint satisfaction against performance. To demonstrate the effectiveness of our approach, we first compare it against state-of-the-art baseline methods in a synthetic environment, highlighting the impact of intrinsic environmental noise across different risk configurations. Finally, we evaluate our framework in a real-world use case involving a 5G mobile network where only our approach satisfies consistently the system constraint (a signal processing reliability target) with a small performance toll (8.5% increase in power consumption).