State-action Space Research Articles

Deep Reinforcement Learning-Assisted Optimization for Resource Allocation in Downlink OFDMA Cooperative Systems.

This paper considers a downlink resource-allocation problem in distributed interference orthogonal frequency-division multiple access (OFDMA) systems under maximal power constraints. As the upcoming fifth-generation (5G) wireless networks are increasingly complex and heterogeneous, it is challenging for resource allocation tasks to optimize the system performance metrics and guarantee user service requests simultaneously. Because of the non-convex optimization problems, using existing approaches to find the optimal resource allocation is computationally expensive. Recently, model-free reinforcement learning (RL) techniques have become alternative approaches in wireless networks to solve non-convex and NP-hard optimization problems. In this paper, we study a deep Q-learning (DQL)-based approach to address the optimization of transmit power control for users in multi-cell interference networks. In particular, we have applied a DQL algorithm for resource allocation to maximize the overall system throughput subject to the maximum power and SINR constraints in a flat frequency channel. We first formulate the optimization problem as a non-cooperative game model, where the multiple BSs compete for spectral efficiencies by improving their achievable utility functions while ensuring the quality of service (QoS) requirements to the corresponding receivers. Then, we develop a DRL-based resource allocation model to maximize the system throughput while satisfying the power and spectral efficiency requirements. In this setting, we define the state-action spaces and the reward function to explore the possible actions and learning outcomes. The numerical simulations demonstrate that the proposed DQL-based scheme outperforms the traditional model-based solution.

Efficient reinforcement learning with least-squares soft Bellman residual for robotic grasping

Grasping control of intelligent robots has to deal with the difficulties of model uncertainties and nonlinearities. In this paper, we propose the Kernel-based Least-Squares Soft Bellman residual Actor–Critic (KLSAC) algorithm for robotic grasping. In the proposed approach, a novel linear temporal-difference learning algorithm using the least-squares soft Bellman residual (LS2BR) method is designed for policy evaluation. In addition, KLSAC adopts a sparse-kernel feature representation method based on approximate linear dependency (ALD) analysis to construct features for continuous state–action space. Compared with typical deep reinforcement learning algorithms, KLSAC has two main advantages: firstly, the critic module has the capacity for rapid convergence by computing the fixed point of the linear soft Bellman equation via the least-squares optimization method. Secondly, the kernel-based features construction approach only requires predefining the basic kernel function and can improve the generalization ability of KLSAC. The simulation studies on robotic grasping control were conducted in the V-REP simulator. The results demonstrate that compared with other typical RL algorithms (e.g., SAC and BMPO), the proposed KLSAC algorithm can achieve better performance in terms of sample efficiency and asymptotic convergence property. Furthermore, experimental results on a real UR5 robot validated that KLSAC performed well in the real world.

Autonomous collision avoidance system in a multi-ship environment based on proximal policy optimization method

To address the challenge of local path planning in a complex, dynamic environment with multiple ships, we built an autonomous collision avoidance system based on deep reinforcement learning in open water. The system switches between path-tracking and collision avoidance modes in real time, during which any collision danger is perceived through encounter identification and risk calculation. Subsequently, the proximal policy optimization method is applied to design the state space, action space, and reward functions. Decision-making offers advantages in terms of continuous action space, perception of global navigation information, compliance with the International Regulations for Preventing Collisions at Sea (COLREGs), and smooth maneuverings. Simulations are performed under multi-ship encounter conditions, considering the ship hydrodynamic model, environmental disturbance force model, COLREGs, and good seamanship. The results indicate the effectiveness and ability of the proposed system to take appropriate action when faced with dangerous situations under rule constraints and diverse scenarios.

Path Planning for Ferry Crossing Inland Waterways Based on Deep Reinforcement Learning

Path planning is a key issue for safe navigation of inland ferries. With the development of ship intelligence, how to enhance the decision–support system of a ferry in a complex navigation environment is one of the key issues. The inland ferries need to cross the channel frequently and, thus, risky encounters with target ships in the waterway are more frequent, so they need an intelligent decision–support system that can deal with complex situations. In this study, a reinforced deep learning method is proposed for path planning of inland ferries during crossing of the waterways. In the study, the state space, action space and reward function of the Deep Q-network (DQN) model are designed and improved to establish an autonomous navigation method for ferries considering both economy and safety. The DQN model also takes into account the crossing behavior, navigation economy and safety. Finally, the model is applied to case studies to verify its effectiveness.

Adaptive Weight Tuning of EWMA Controller via Model-Free Deep Reinforcement Learning

Exponentially weighted moving average (EWMA) controllers have been extensively studied for run-to-run (RtR) control in semiconductor manufacturing processes. However, the EWMA controller with a fixed weight struggles to achieve excellent performance under unknown stochastic disturbances. To improve the performance of EMWA via online parameter tuning, an intelligent strategy using deep reinforcement learning (DRL) technique is developed in this work. To begin with, the weight adjusting problem is established as a Markov decision process. Meanwhile, simple state space, action space and reward function are designed. Then, the classical deep deterministic policy gradient (DDPG) algorithm is utilized to adjust the weight online. Moreover, a quantile regression-based DDPG (QR-DDPG) algorithm is further verified the effectiveness of the proposed method. Finally, the developed scheme is implemented on a deep reactive ion etching process. Comparisons are conducted to show the superiority of the presented approach in terms of disturbance rejection and target tracking.

Multiagent Soft Actor–Critic for Traffic Light Timing

Deep reinforcement learning has strong perception and decision-making capabilities that can effectively solve the problem of continuous high-dimensional state-action space and has become the mainstream method in the field of traffic light timing. However, due to model structural defects or different strategic mechanisms of models, most deep reinforcement learning models have problems such as convergence and divergence or poor exploration capabilities. Therefore, this paper proposes a multi-agent Soft Actor–Critic (SAC) for traffic light timing. Multi-agent SAC adds an entropy item to measure the randomness of the strategy in the objective function of traditional reinforcement learning and maximizes the sum of expected reward and entropy item to improve the model’s exploration ability. The system model can learn multiple optimal timing schemes, avoid repeated selection of the same optimal timing scheme and fall into a local optimum or fail to converge. Meanwhile, it abandons low reward value strategies to reduce data storage and sampling complexity, accelerate training, and improve the stability of the system. Comparative experiments show that the method based on multi-agent SAC traffic light timing can solve the existing problems of deep reinforcement learning and improve the efficiency of vehicles passing through in different traffic scenarios.

A Centralized Routing for Lifetime and Energy Optimization in WSNs Using Genetic Algorithm and Least-Square Policy Iteration

Q-learning has been primarily used as one of the reinforcement learning (RL) techniques to find the optimal routing path in wireless sensor networks (WSNs). However, for the centralized RL-based routing protocols with a large state space and action space, the baseline Q-learning used to implement these protocols suffers from degradation in the convergence speed, network lifetime, and network energy consumption due to the large number of learning episodes required to learn the optimal routing path. To overcome these limitations, an efficient model-free RL-based technique called Least-Square Policy Iteration (LSPI) is proposed to optimize the network lifetime and energy consumption in WSNs. The resulting designed protocol is a Centralized Routing Protocol for Lifetime and Energy Optimization with a Genetic Algorithm (GA) and LSPI (CRPLEOGALSPI). Simulation results show that the CRPLEOGALSPI has improved performance in network lifetime and energy consumption compared to an existing Centralized Routing Protocol for Lifetime Optimization with GA and Q-learning (CRPLOGARL). This is because the CRPLEOGALSPI chooses a routing path in a given state considering all the possible routing paths, and it is not sensitive to the learning rate. Moreover, while the CRPLOGARL evaluates the optimal policy from the Q-values, the CRPLEOGALSPI updates the Q-values based on the most updated information regarding the network dynamics using weighted functions.

Path Planning of Unmanned Aerial Vehicle in Complex Environments Based on State-Detection Twin Delayed Deep Deterministic Policy Gradient

This paper investigates the path planning problem of an unmanned aerial vehicle (UAV) for completing a raid mission through ultra-low altitude flight in complex environments. The UAV needs to avoid radar detection areas, low-altitude static obstacles, and low-altitude dynamic obstacles during the flight process. Due to the uncertainty of low-altitude dynamic obstacle movement, this can slow down the convergence of existing algorithm models and also reduce the mission success rate of UAVs. In order to solve this problem, this paper designs a state detection method to encode the environmental state of the UAV’s direction of travel and compress the environmental state space. In considering the continuity of the state space and action space, the SD-TD3 algorithm is proposed in combination with the double-delayed deep deterministic policy gradient algorithm (TD3), which can accelerate the training convergence speed and improve the obstacle avoidance capability of the algorithm model. Further, to address the sparse reward problem of traditional reinforcement learning, a heuristic dynamic reward function is designed to give real-time rewards and guide the UAV to complete the task. The simulation results show that the training results of the SD-TD3 algorithm converge faster than the TD3 algorithm, and the actual results of the converged model are better.

Classical Actor-Critic Applied to the Control of a Self – Regulatory Process

The industrial adoption of Reinforcement Learning (RL) is currently hindered by an RL agent's potentially erratic learning behaviour, black-box decision-making, and the need for problem-specific tuning. Considering these shortcomings, an Actor-Critic RL agent is applied within the context of behavioural cloning where an initial policy estimate (the warm-starting policy) is provided before RL adaptation. A PI controller is equivalent to a deterministic policy acting on the state space defined by the error and integral error; thus, a PI controller may be used as the warm-starting policy. Two policy types are proposed: the additive- and the emulating policy type. The additive policy was generated by adding the output from a neural network to the plane defined by a PI controller in the state-action space. The emulating policy was constructed by fitting a neural network to emulate the same PI controller before policy adaptation through RL. The results illustrate inverting gradients to be potentially useful as a safeguard against policy divergence resulting from valve saturation combined with inaccurate state-value estimates. It is further shown that performance improvements can be obtained while maintaining desired properties of the original policy (i.e., PI controller) through application of the Actor-Critic algorithm. Finally, RL is contextualised as an inherent connection between classical and optimal control. By constructing a PI control policy describing the selection of the control input as a function of feedback error and the integral thereof, a graphical representation of the subsequent RL adaptations made is obtained, thus enabling interpretability of the resulting RL controller.

Joint Scheduling of Proactive Pushing and On-Demand Transmission Over Shared Spectrum for Profit Maximization

Proactive pushing has emerged as a promising solution to scale the service capacity by utilizing idle spectrum resources when on-demand transmission cannot fully exploit the spectrum resources during off-peak times. How to schedule both pushing and on-demand transmission jointly to meet a higher spectrum efficiency becomes a critical issue. Moreover, efficient and fair spectrum sharing among multiple resource schedulers remains open. In this paper, we introduce virtual network operators (VNOs) as schedulers that pay for consumed bandwidth and jointly schedule pushing and on-demand services. We adopt nonlinear spectrum pricing schemes with a convex and increasing price function enabling each VNO to share spectrum resources appropriately and independently. Considering the revenue from users and the cost for spectrum, we formulate a Markov decision process (MDP) to maximize the profit of VNO. A modified value iteration algorithm is applied to solve the MDP with reduced computational complexity. Furthermore, we show the structure of the optimal policy and provide the upper and lower bounds for the optimal performance. We present a low-complexity heuristic policy that can scale in practice with large state spaces and action spaces.

Deep Reinforcement Learning Based Active Pantograph Control Strategy in High-Speed Railway

The pantograph-catenary system (PCS) is the essential power supply system in the high-speed railway, but its coupling performance is influenced significantly by the rapidly increasing train speed. The actively controlled pantograph is one of the promising technologies to suppress the fluctuation of the pantograph-catenary contact force (PCCF). In this paper, we propose a novel pantograph control strategy based on deep reinforcement learning (DRL) to overcome the complex time-varying characteristic of PCS, which distorts the system identification of the classical control methods. First, a non-linear pantograph-catenary system model is established based on the finite element and multi-body dynamics theory as the simulation environment in DRL. Then, the state space, action space, and reward in DRL are redesigned to train the agent, which is suitable for PCS. Finally, the effectiveness and robustness of our proposed method are verified under various working conditions and parameter disturbances. The experiment results show that our control strategy can reduce the PCCF fluctuation up to 40% and reject parametric perturbation while achieving state-of-the-art performance on the benchmarks.

PPO-Based PDACB Traffic Control Scheme for Massive IoV Communications

Traffic control is regarded as a key issue to alleviate congestion in internet of vehicles (IoV) machine-type communications (MTC). Recently, many traffic control schemes have been studied, such as access class barring (ACB) scheme and back-off (BO) scheme. However, the dynamics of traffic and the heterogeneous requirements of different IoV applications are not considered in most existing studies, which is significant for the random access resource allocation. In this paper, we consider a hybrid scheme, combining the priority dynamic ACB (PDACB) scheme and BO scheme. The IoV devices are classified depending on different delay characteristics, where the delay-sensitive devices are classified as high priority. The target is to maximum the successful transmission of packets with the success rate constraint by adjusting the various ACB factors. Proximal policy optimization (PPO) algorithm as a unique deep reinforcement learning (DRL) method is utilized in this paper, which can obtain continuous action space and solve for the optimal ACB factors without estimating backlog of nodes. A quick convergence is achieved by designing sensible state space, action space and reward. The access capability of the PDACB traffic control scheme is verified by simulations.

A Novel Exploration-Exploitation-Based Adaptive Law for Intelligent Model-Free Control Approaches.

Model-free control approaches require advanced exploration-exploitation policies to achieve practical tasks such as learning to bipedal robot walk in unstructured environments. In this article, we first construct a comprehensive exploration-exploitation policy that carries quality knowledge about the long-term predictor and the control policy, and the control signal of the model-free algorithms. Therefore, the developed model-free algorithm continues exploration by adjusting its unknown parameters until the desired learning and control are accomplished. Second, we provide an utterly model-free adaptive law enriched with the exploration-exploitation policy and derived step-by-step using the exact analogy of the model-based solution. The obtained adaptive control law considers the control signal saturation and the control signal (input) delay. Performed Lyapunov stability analysis ensures the convergence of the adaptive law that can also be used for intelligent control approaches. Third, we implement the adaptive algorithm in real time on a challenging benchmark system: a fourth-order, coupled dynamics, input saturated, and time-delayed underactuated manipulator. The results show that the proposed adaptive algorithm explores larger state-action spaces and treats the vanishing gradient problem in both learning and control. Also, we notice from the results that the learning and control properties of the adaptive algorithm are optimized as required.

Congestion-Aware Path Coordination Game With Markov Decision Process Dynamics

Inspired by the path coordination problem arising from robo-taxis, warehouse management, and mixed-vehicle routing, we model a group of heterogeneous players responding to stochastic demands as a congestion game under Markov decision process dynamics. Players share a common state-action space but have unique transition dynamics, and each player’s unique cost is a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">function</i> of the joint state-action probability distribution. For a class of player cost functions, we formulate the player-specific optimization problem, prove equivalence between the Nash equilibrium and the solution of a potential minimization problem, and derive dynamic programming approaches to solve the Nash equilibrium. We apply this game to model multi-agent path coordination and introduce congestion-based cost functions that enable players to complete individual tasks while avoiding congestion with their opponents. Finally, we present a learning algorithm for finding the Nash equilibrium that has linear complexity in the number of players. We demonstrate our game model on a multi-robot warehouse path coordination problem, in which robots autonomously retrieve and deliver packages while avoiding congested paths.

Intra-Domain Knowledge Reuse Assisted Reinforcement Learning for Fast Anti-Jamming Communication

The novel technology based on reinforcement learning (RL) is considered as a promising direction to achieve cognitive and even intelligent anti-jamming communication. However, traditional RL converges slowly in the face of huge state-action space, while the performance improvement of deep RL depends on high-cost computing resources. Under unknown and dynamic malicious jamming attacks, it is still a major challenge to realize online anti-jamming communication through fast and autonomous learning. In this paper, we propose a fast anti-jamming scheme based on intra-domain knowledge reuse for distributed wireless networks. First, we introduce the Bisimulation Relation to measure the similarity of different state-action pairs in the communication anti-jamming problem, so as to build a bridge of knowledge reuse between state-action pairs. This similarity stems from the fact that the anti-jamming problem can be divided into two categories according to whether it successfully resists jamming attacks, in which different state-action pairs in the same category have similarities. Then, through the real-time update and reuse of state-action values, each node can independently learn the characteristics of dynamic environment and adjust the transmission strategy quickly, so as to avoid external malicious jamming and internal mutual interference. The simulation results show that our proposed scheme significantly accelerates the convergence speed and improves the network normalized throughput compared with the benchmark against unknown and dynamic jamming, and has remarkable agility and jamming memory.

A Hybrid Linear Programming-Reinforcement Learning Method for Optimal Energy Hub Management

Reinforcement learning (RL) is a subset of artificial intelligence in which a decision-making agent tries to act optimally in an environment by controlling different parameters. There is no need to identify and mathematically formulate the environmental constraints in such a method. Moreover, the RL agent does not need prior information about future outcomes to act optimally in the current situation. However, its performance is adversely affected by the environmental complexity, which increases the agent’s effort to choose the optimal action in a particular condition. Integrating RL and linear programming (LP) methods is beneficial to tackle this problem as it reduces the state-action space that the agent should learn. In this regard, the optimization variables are divided into two categories. First, experience-dependent variables which have an inter-time dependency, and their values depend on the agent’s decision. Second, experience-independent variables whose values depend on the LP model and have no inter-time connection. The numerical results of integrating mentioned methods have demonstrated the hybrid model’s effectiveness in converging to the global optimum with more than 95% accuracy.

Reinforcement learning using Deep Q networks and Q learning accurately localizes brain tumors on MRI with very small training sets

BackgroundSupervised deep learning in radiology suffers from notorious inherent limitations: 1) It requires large, hand-annotated data sets; (2) It is non-generalizable; and (3) It lacks explainability and intuition. It has recently been proposed that reinforcement learning addresses all three of these limitations. Notable prior work applied deep reinforcement learning to localize brain tumors with radiologist eye tracking points, which limits the state-action space. Here, we generalize Deep Q Learning to a gridworld-based environment so that only the images and image masks are required.MethodsWe trained a Deep Q network on 30 two-dimensional image slices from the BraTS brain tumor database. Each image contained one lesion. We then tested the trained Deep Q network on a separate set of 30 testing set images. For comparison, we also trained and tested a keypoint detection supervised deep learning network on the same set of training/testing images.ResultsWhereas the supervised approach quickly overfit the training data and predictably performed poorly on the testing set (11% accuracy), the Deep Q learning approach showed progressive improved generalizability to the testing set over training time, reaching 70% accuracy.ConclusionWe have successfully applied reinforcement learning to localize brain tumors on 2D contrast-enhanced MRI brain images. This represents a generalization of recent work to a gridworld setting naturally suitable for analyzing medical images. We have shown that reinforcement learning does not over-fit small training sets, and can generalize to a separate testing set.

Prioritized Hindsight with Dual Buffer for Meta-Reinforcement Learning

Sharing prior knowledge across multiple robotic manipulation tasks is a challenging research topic. Although the state-of-the-art deep reinforcement learning (DRL) algorithms have shown immense success in single robotic tasks, it is still challenging to extend these algorithms to be applied directly to resolve multi-task manipulation problems. This is mostly due to the problems associated with efficient exploration in high-dimensional state and continuous action spaces. Furthermore, in multi-task scenarios, the problem of sparse reward and sample inefficiency of DRL algorithms is exacerbated. Therefore, we propose a method to increase the sample efficiency of the soft actor-critic (SAC) algorithm and extend it to a multi-task setting. The agent learns a prior policy from two structurally similar tasks and adapts the policy to a target task. We propose a prioritized hindsight with dual experience replay to improve the data storage and sampling technique, which, in turn, assists the agent in performing structured exploration that leads to sample efficiency. The proposed method separates the experience replay buffer into two buffers to contain real trajectories and hindsight trajectories to reduce the bias introduced by the hindsight trajectories in the buffer. Moreover, we utilize high-reward transitions from previous tasks to assist the network in easily adapting to the new task. We demonstrate the proposed method based on several manipulation tasks using a 7-DoF robotic arm in RLBench. The experimental results show that the proposed method outperforms vanilla SAC in both a single-task setting and multi-task setting.

Operation strategy optimization of combined cooling, heating, and power systems with energy storage and renewable energy based on deep reinforcement learning

Combined cooling, heating, and power (CCHP), coupled with renewable energy generation and energy storage can achieve a low-carbon, multi-energy complementary, and flexible energy system. However, the inclusion of renewable resources and energy storage poses significant challenges to the operational management of such systems. Conventional algorithms are limited when solving nonlinear and uncertain non-convex optimization problems thus, deep reinforcement learning (DRL) is considered the most effective method to solve these issues because of its powerful nonlinear fitting ability, model-free utilization, and capability of solving decision-making problems. This study proposes a novel operation strategy optimization model based on DRL to minimize the operation cost of an energy system, which composes of CCHP, photovoltaic generation and energy storage system. The optimization problem was transformed into a Markov decision process (MDP), and the state space and action space are differently modeling in summer and winter scenarios. Two models of handling action constraints are proposed and testing in terms of performance. Two DRL algorithms, Deep Deterministic policy gradient (DDPG) and Twin Delayed Deep Deterministic policy gradient (TD3), were verified separately via comparison with the conventional algorithms, particle swarm optimization (PSO) and mathematical programming under perfect input conditions. Results show that setting different training environments for summer and winter respectively can get better optimization results, and the model with better performance in handling action constraints was validated from the operation strategy and optimization results. The performance of the TD3 method is comparable to the theoretical benchmark, with an average error of approximately 5%. The computation time for a single-step online decision is only 0.001s and for a 24-step online decision is only 0.006s, it significantly improves operational efficiency, demonstrating the adaptability of DRL methods for optimization and computational performance.

Model-free inverse reinforcement learning with multi-intention, unlabeled, and overlapping demonstrations

In this paper, we define a novel inverse reinforcement learning (IRL) problem where the demonstrations are multi-intention, i.e., collected from multi-intention experts, unlabeled, i.e., without intention labels, and partially overlapping, i.e., shared between multiple intentions. In the presence of overlapping demonstrations, current IRL methods, developed to handle multi-intention and unlabeled demonstrations, cannot successfully learn the underlying reward functions. To solve this limitation, we propose a novel clustering-based approach to disentangle the observed demonstrations and experimentally validate its advantages. Traditional clustering-based approaches to multi-intention IRL, which are developed on the basis of model-based Reinforcement Learning (RL), formulate the problem using parametric density estimation. However, in high-dimensional environments and unknown system dynamics, i.e., model-free RL, the solution of parametric density estimation is only tractable up to the density normalization constant. To solve this, we formulate the problem as a mixture of logistic regressions to directly handle the unnormalized density. To research the challenges faced by overlapping demonstrations, we introduce the concepts of shared pair, which is a state-action pair that is shared in more than one intention, and separability, which resembles how well the multiple intentions can be separated in the joint state-action space. We provide theoretical analyses under the global optimality condition and the existence of shared pairs. Furthermore, we conduct extensive experiments on four simulated robotics tasks, extended to accept different intentions with specific levels of separability, and a synthetic driver task developed to directly control the separability. We evaluate the existing baselines on our defined problem and demonstrate, theoretically and experimentally, the advantages of our clustering-based solution, especially when the separability of the demonstrations decreases.