Action Selection Strategy Research Articles

Deep Reinforcement Learning Based Link Adaptation Technique for LTE/NR Systems

Outdated channel quality indicator (CQI) feedback causes severe performance degradation of traditional link adaptation (LA) techniques in long term evolution (LTE) and new radio (NR) systems. This paper puts forth a deep reinforcement learning (DRL) based link adaptation (LA) technique, referred to as deep reinforcement learning link adaptation (DRLLA), to select efficient modulation and coding scheme (MCS) in the presence of the outdated CQI feedback. The goal of DRLLA is to maximize the link throughput while achieving a low block error rate (BLER). We first give explicit definitions of state, action, and reward in DRL paradigms, thereby realizing DRLLA. Then, to trade off the throughput against the BLER, we further develop a new experience replay mechanism called classified experience replay (CER) as the underpinning technique in DRLLA. In CER, experiences are separated into two buckets, one for successful experiences and the other for failed experiences, and then a fixed proportion from each is sampled to replay. The essence of CER is to obtain different trade-offs via adjusting the proportion among different training experiences. Furthermore, to reduce the signaling overhead and the system reconfiguration cost caused by frequent MCS switching, we propose a new action selection strategy termed as switching-controlled <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy (SC- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy) for DRLLA. Simulation results demonstrate that compared with the state-of-the-art OLLA, LTSLA, and DRLLA with other experience replay mechanisms, DRLLA with CER can achieve higher throughput and lower BLER in various time-varying scenarios, and be more robust to different CQI feedback delays and CQI reporting periods. Furthermore, with the SC- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -greedy policy, DRLLA can capture better trade-offs between the link transmission quality and the MCS switching overhead compared with other baselines.

A Quantum States Preparation Method Based on Difference-Driven Reinforcement Learning

Due to the large state space of the two-qubit system, and the adoption of ladder reward function in the existing quantum state preparation methods, the convergence speed is slow and it is difficult to prepare the desired target quantum state with high fidelity under limited conditions. To solve the above problems, a difference-driven reinforcement learning (RL) algorithm for quantum state preparation of two-qubit system is proposed by improving the reward function and action selection strategy. Firstly, a model is constructed for the problem of preparing quantum states of a two-qubit system, with restrictions on the type of quantum gates and the time for quantum state evolution. In the preparation process, a weighted differential dynamic reward function is designed to assist the algorithm quickly obtain the maximum expected cumulative reward. Then, an adaptive [Formula: see text]-greedy action selection strategy is adopted to achieve a balance between exploration and utilization to a certain extent, thereby improving the fidelity of the final quantum state. The simulation results show that the proposed algorithm can prepare quantum state with high fidelity under limited conditions. Compared with other algorithms, it has different degrees of improvement in convergence speed and fidelity of the final quantum state.

Astrocyte activities in the external globus pallidus regulate action-selection strategies in reward-seeking behaviors

Astrocyte activities in the external globus pallidus regulate action-selection strategies in reward-seeking behaviors

A Path Planning Algorithm for Mobile Robot Based on MR-QL in the Unknown Scene

The Q-Learning (QL) algorithm is widely used for path planning. As the scene in which the mobile robot is located becomes complex, the algorithm suffers from the limitations of low convergence speed and long exploration paths. Therefore, a Max Reward-Q-learning (MR-QL) path planning algorithm based on maximum reward is proposed for complex unknown scenarios. The original algorithm’s discrete reward function and action selection strategy are improved, and a new reward function is designed to dynamically adjust the reward mechanism to heuristically guide the robot motion. The action selection strategy is also optimized to avoid invalid exploration and improve the algorithm’s convergence. Finally, three experimental environments with different complexity are constructed to demonstrate the feasibility of the proposed algorithm. The simulation results show that the MR-QL algorithm is about 50% of the original algorithm in terms of exploration step length and training time, and the convergence speed of the algorithm is better than the original algorithm.

Early life adversity impaired dorsal striatal synaptic transmission and behavioral adaptability to appropriate action selection in a sex-dependent manner.

Early life adversity (ELA) is a major health burden in the United States, with 62% of adults reporting at least one adverse childhood experience. These experiences during critical stages of brain development can perturb the development of neural circuits that mediate sensory cue processing and behavioral regulation. Recent studies have reported that ELA impaired the maturation of dendritic spines on neurons in the dorsolateral striatum (DLS) but not in the dorsomedial striatum (DMS). The DMS and DLS are part of two distinct corticostriatal circuits that have been extensively implicated in behavioral flexibility by regulating and integrating action selection with the reward value of those actions. To date, no studies have investigated the multifaceted effects of ELA on aspects of behavioral flexibility that require alternating between different action selection strategies or higher-order cognitive processes, and the underlying synaptic transmission in corticostriatal circuitries. To address this, we employed whole-cell patch-clamp electrophysiology to assess the effects of ELA on synaptic transmission in the DMS and DLS. We also investigated the effects of ELA on the ability to update action control in response to outcome devaluation in an instrumental learning paradigm and reversal of action-outcome contingency in a water T-maze paradigm. At the circuit level, ELA decreased corticostriatal glutamate transmission in male but not in female mice. Interestingly, in DMS, glutamate transmission is decreased in male ELA mice, but increased in female ELA mice. ELA impaired the ability to update action control in response to reward devaluation in a context that promotes goal-directedness in male mice and induced deficits in reversal learning. Overall, our findings demonstrate the sex- and region-dependent effects of ELA on behavioral flexibility and underlying corticostriatal glutamate transmission. By establishing a link between ELA and circuit mechanisms underlying behavioral flexibility, our findings will begin to identify novel molecular mechanisms that can represent strategies for treating behavioral inflexibility in individuals who experienced early life traumatic incidents.

Towards end-to-end formation control for robotic fish via deep reinforcement learning with non-expert imitation

Collaboration of multiple robotic fish can accomplish various underwater tasks effectively. However, controlling robotic fish to maintain a specific formation remains a huge challenge, especially in complex and changing flow fields. This paper presents an end-to-end formation control approach in the leader–follower topology by combining deep reinforcement learning and imitation learning. First, we build a high-fidelity environment based on computational fluid dynamics (CFD) to generate samples for training the formation controller. In this environment, we maneuver the robotic fish by adjusting the maximum swing of its tail. Then, we model the formation control problem as a Markov decision process (MDP), where a compound reward function is tailored to guide the training. To improve the learning efficiency of the deep reinforcement learning (DRL) based controller, we propose a novel DRL algorithm on the top of deep Q-networks (DQN) and behavior cloning, which we call dueling double DQN (D3QN) with imitation. Combining with the designed imitation-based action selection strategy, this algorithm significantly reduce the blindness of agent exploration at the beginning of training. A series of experiments demonstrate the advantages of the proposed algorithm in terms of control accuracy, training efficiency, as well as generalization ability for different formation configurations.

Reinforcement learning with dynamic completion for answering multi-hop questions over incomplete knowledge graph

Text-enhanced and implicit reasoning methods are proposed for answering questions over incomplete knowledge graph (KG), whereas prior studies either rely on external resources or lack necessary interpretability. This article desires to extend the line of reinforcement learning (RL) methods for better interpretability and dynamically augment original KG action space with additional actions. To this end, we propose a RL framework along with a dynamic completion mechanism, namely Dynamic Completion Reasoning Network (DCRN). DCRN consists of an action space completion module and a policy network. The action space completion module exploits three sub-modules (relation selector, relation pruner and tail entity predictor) to enrich options for decision making. The policy network calculates probability distribution over joint action space and selects promising next-step actions. Simultaneously, we employ the beam search-based action selection strategy to alleviate delayed and sparse rewards. Extensive experiments conducted on WebQSP, CWQ and MetaQA demonstrate the effectiveness of DCRN. Specifically, under 50% KG setting, the Hits@1 performance improvements of DCRN on MetaQA-1H and MetaQA-3H are 2.94% and 1.18% respectively. Moreover, under 30% and 10% KG settings, DCRN prevails over all baselines by 0.9% and 1.5% on WebQSP, indicating the robustness to sparse KGs.

Path Planning Algorithm for Unmanned Surface Vessel Based on Multiobjective Reinforcement Learning.

It is challenging to perform path planning tasks in complex marine environments as the unmanned surface vessel approaches the goal while avoiding obstacles. However, the conflict between the two subtarget tasks of obstacle avoidance and goal approaching makes the path planning difficult. Thus, a path planning method for unmanned surface vessel based on multiobjective reinforcement learning is proposed under the complex environment with high randomness and multiple dynamic obstacles. Firstly, the path planning scene is set as the main scene, and the two subtarget scenes including obstacle avoidance and goal approaching are divided from it. The action selection strategy in each subtarget scene is trained through the double deep Q-network with prioritized experience replay. A multiobjective reinforcement learning framework based on ensemble learning is further designed for policy integration in the main scene. Finally, by selecting the strategy from subtarget scenes in the designed framework, an optimized action selection strategy is trained and used for the action decision of the agent in the main scene. Compared with traditional value-based reinforcement learning methods, the proposed method achieves a 93% success rate in path planning in simulation scenes. Furthermore, the average length of the paths planned by the proposed method is 3.28% and 1.97% shorter than that of PER-DDQN and dueling DQN, respectively.

Multi-Node Joint Power Allocation Algorithm Based on Hierarchical Game Learning in Underwater Acoustic Sensor Networks

In order to improve the overall service quality of the network and reduce the level of network interference, power allocation has become one of the research focuses in the field of underwater acoustic communication in recent years. Aiming at the issue of power allocation when channel information is difficult to obtain in complex underwater acoustic communication networks, a completely distributed game learning algorithm is proposed that does not require any prior channel information and direct information exchange between nodes. Specifically, the power allocation problem is constructed as a multi-node multi-armed bandit (MAB) game model. Then, considering nodes as agents and multi-node networks as multi-agent networks, a power allocation algorithm based on a softmax-greedy action selection strategy is proposed. In order to improve the learning efficiency of the agent, reduce the learning cost, and mine the historical reward information, a learning algorithm based on the two-layer hierarchical game learning (HGL) strategy is further proposed. Finally, the simulation results show that the algorithm not only shows good convergence speed and stability but also can adapt to a harsh and complex network environment and has a certain tolerance for incomplete channel information acquisition.

Risk-Aware Travel Path Planning Algorithm Based on Reinforcement Learning during COVID-19

The outbreak of COVID-19 brought great inconvenience to people’s daily travel. In order to provide people with a path planning scheme that takes into account both safety and travel distance, a risk aversion path planning model in urban traffic scenarios was established through reinforcement learning. We have designed a state and action space of agents in a “point-to-point” way. Moreover, we have extracted the road network model and impedance matrix through SUMO simulation, and have designed a Restricted Reinforcement Learning-Artificial Potential Field (RRL-APF) algorithm, which can optimize the Q-table initialization operation before the agent learning and the action selection strategy during learning. The greedy coefficient is dynamically adjusted through the improved greedy strategy. Finally, according to different scenarios, our algorithm is verified by the road network model and epidemic historical data in the surrounding areas of Xinfadi, Beijing, China, and comparisons are made with common Q-Learning, the Sarsa algorithm and the artificial potential field-based reinforcement learning (RLAFP) algorithm. The results indicate that our algorithm improves convergence speed by 35% on average and the travel distance is reduced by 4.3% on average, while avoiding risk areas, compared with the other three algorithms.

DRL-M4MR: An intelligent multicast routing approach based on DQN deep reinforcement learning in SDN

Traditional multicast routing methods have some problems in constructing a multicast tree. These problems include limited access to network state information, poor adaptability to dynamic and complex changes in the network, and inflexible data forwarding. To address these defects, the optimal multicast routing problem in software-defined networking (SDN) is tailored as a multiobjective optimization problem, and DRL-M4MR, an intelligent multicast routing algorithm based on the deep Q network (DQN) deep reinforcement learning (DRL) method is designed to construct a multicast tree in a software-defined network. First, combining the characteristics of SDN global network-aware information, the multicast tree state matrix, link bandwidth matrix, link delay matrix and link packet loss rate matrix are designed as the state space of the reinforcement learning agent to solve the problem in that the original method cannot make full use of network status information. Second, the action space of the agent is all the links in the network, and the action selection strategy is designed to add the links to the current multicast tree in four cases. Third, single-step and final reward function forms are designed to guide the agent to make decisions to construct the optimal multicast tree. The double network architectures, dueling network architectures and prioritized experience replay are adopted to improve the learning efficiency and convergence of the agent. Finally, after the DRL-M4MR agent is trained, the SDN controller installs the multicast flow entries by reversely traversing the multicast tree to the SDN switches to implement intelligent multicast routing. The experimental results show that, compared with existing algorithms, the multicast tree constructed by DRL-M4MR can obtain better bandwidth, delay, and packet loss rate performance after training, and it can make more intelligent multicast routing decisions in a dynamic network environment. Code and DRL model are available at https://github.com/GuetYe/DRL-M4MR.

Coordinated intelligent control of the flight control system and shape change of variable sweep morphing aircraft based on dueling-DQN

This work develops a method of coordinated control between the flight control system and the morphing mechanism for a variable sweep morphing aircraft. A variable sweep morphing aircraft model with six degrees of freedom is established and separated into cascaded loops. Based on the principle of multi-time scale separation, the proportional-integral control, nonlinear dynamic inversion, and incremental nonlinear dynamic inversion methods are employed to design a flight control system as the basic controller to manipulate the control surfaces and engine thrust. Considering the difficulty of designing a morphing controller based on a model-based approach and the requirement of optimizing the flight performance through morphing, we design a dueling-deep Q network learning-based intelligent morphing controller. Further, we take the basic controller and dynamic model of the aircraft as the environments in the framework of reinforcement learning. The time step of the agent is extended to solve the convergence problem caused by the short control cycle. Furthermore, the state space, action selection strategy, and performance index function are properly designed. The results from the simulations validate the effectiveness of the proposed scheme.

Reversible Data Hiding for Color Images Based on Adaptive 3D Prediction-Error Expansion and Double Deep Q-Network

Reversible data hiding (RDH) for color images has attracted increasing attention in recent years. Due to its effective utilization of the correlation between prediction errors, high-dimensional prediction-error expansion (PEE) can achieve much better performance for color image RDH than low-dimensional PEE. However, existing studies only focus on high-dimensional PEE with nonadaptive embedding. To further improve the embedding performance for color images, we propose a novel three-dimensional PEE method that is adaptive to image content. Double deep Q-network (DDQN), introduced to RDH for the first time, is adopted to find the optimal mapping paths for PEE. In addition, an action selection scheme is presented for DDQN to efficiently find the reversible mapping paths. Extensive experiments show that the proposed method outperforms existing color image RDH methods in image quality.

Real-Time Data Transmission Scheduling Algorithm for Wireless Sensor Networks Based on Deep Q-Learning

In the industrial environment, the data transmission of Wireless Sensor Networks (WSNs) usually has strict deadline requirements. Improving the reliability and real-time performance of data transmission has become one of the critical issues in WSNs research. One of the main methods to improve the network performance of WSNs is to schedule the transmission process. An effective scheduling algorithm can meet the requirements of a strict industrial environment for network performance, which is of great research significance. Aiming at the problem of concurrent data transmission in WSNs, a real-time data transmission scheduling algorithm based on deep Q-learning is proposed. The algorithm comprehensively considers the influence of the remaining deadline, remaining hops, and unassigned time-slot nodes in the data transmission process, defines the reward function and action selection strategy of Q-learning, and guides the system state information transfer process. At the same time, deep learning and Q-learning are combined to solve the problem of disaster maintenance caused by the large scale of the system state. A multi-layer Stacked Auto Encoder (SAE) network model establishes the state-action mapping relationship, and the Q-learning algorithm updates it. Finally, according to the trained SAE network model, the data transmission scheduling strategy of the system in different states is obtained. The network performance of the proposed data transmission scheduling algorithm is analyzed and evaluated by simulation experiments. The simulation results show that compared with the commonly used heuristic algorithms, the proposed algorithm improves real-time performance and can better meet the data transmission requirements of high reliability and real-time WSNs.

Path Planning Method of Mobile Robot Using Improved Deep Reinforcement Learning

A mobile robot path planning method based on improved deep reinforcement learning is proposed. First, in order to conform to the actual kinematics model of the robot, the continuous environmental state space and discrete action state space are designed. In addition, an improved deep Q-network (DQN) method is proposed, which takes the directly collected information as the training samples and combines the environmental state characteristics of the robot and the target point to be reached as the input of the network. DQN method takes the Q value at the current position as the output of the network model and uses ε -greedy strategy for action selection. Finally, the reward function combined with the artificial potential field method is designed to optimize the state-action space. The reward function solves the problem of sparse reward in the environmental state space and makes the action selection of the robot more accurate. Experiments show that compared with the classical DQN method, the average loss function value is reduced by 36.87% and the average reward value is increased by 12.96%, which can effectively improve the working efficiency of mobile robot.

Deep Reinforcement Learning of Collision-Free Flocking Policies for Multiple Fixed-Wing UAVs Using Local Situation Maps

The evolution of artificial intelligence and Internet of Things (IoT) envision a highly integrated artificial IoT (AIoT) network. Flocking and cooperation with multiple unmanned aerial vehicles (UAVs) are expected to play a vital role in industrial AIoT networks. In this article, we formulate the collision-free flocking problem of fixed-wing UAVs as a Markov decision process and solve it in the deep reinforcement learning (DRL) framework. Our method can deal with a variable number of followers by encoding the dynamic environmental state into a fixed-length embedding tensor. Specifically, each follower constructs a fixed-size local situation map that describes the collision risks with other followers nearby. The local situation maps are used by a proposed DRL algorithm to learn the collision-free flocking behavior. To further improve the learning efficiency, we design a reference-point-based action selection strategy and an adaptive mechanism. We compare the proposed MA2D3QN algorithm with several benchmark DRL algorithms through numerical simulation, and we verify its advantages in learning efficiency and performance. Finally, we demonstrate the scalability and adaptability of MA2D3QN in a semiphysical simulation experiment.

Revisiting the Role of the Medial Temporal Lobe in Motor Learning.

Classic taxonomies of memory distinguish explicit and implicit memory systems, placing motor skills squarely in the latter branch. This assertion is in part a consequence of foundational discoveries showing significant motor learning in amnesics. Those findings suggest that declarative memory processes in the medial temporal lobe (MTL) do not contribute to motor learning. Here, we revisit this issue, testing an individual (L. S. J.) with severe MTL damage on four motor learning tasks and comparing her performance to age-matched controls. Consistent with previous findings in amnesics, we observed that L. S. J. could improve motor performance despite having significantly impaired declarative memory. However, she tended to perform poorly relative to age-matched controls, with deficits apparently related to flexible action selection. Further supporting an action selection deficit, L. S. J. fully failed to learn a task that required the acquisition of arbitrary action-outcome associations. We thus propose a modest revision to the classic taxonomic model: Although MTL-dependent memory processes are not necessary for some motor learning to occur, they play a significant role in the acquisition, implementation, and retrieval of action selection strategies. These findings have implications for our understanding of the neural correlates of motor learning, the psychological mechanisms of skill, and the theory of multiple memory systems.

Improved reinforcement learning algorithm for mobile robot path planning

In order to solve the problem that traditional Q-learning algorithm has a large number of invalid iterations in the early convergence stage of robot path planning, an improved reinforcement learning algorithm is proposed. Firstly, the gravitational potential field in the improved artificial potential field algorithm is introduced when the Q table is initialized to accelerate the convergence. Secondly, the Tent Chaotic Mapping algorithm is added to the initial state determination process of the algorithm, which allows the algorithm to explore the environment more fully. In addition, an ε-greed strategy with the number of iterations changing the ε value becomes the action selection strategy of the algorithm, which improves the performance of the algorithm. Finally, the grid map simulation results based on MATLAB show that the improved Q-learning algorithm has greatly reduced the path planning time and the number of non-convergence iterations compared with the traditional algorithm.

Generalized Monte-Carlo Tree Search Extensions for General Game Playing

General Game Playing (GGP) agents must be capable of playing a wide variety of games skillfully. Monte-Carlo Tree Search (MCTS) has proven an effective reasoning mechanism for this challenge, as is reflected by its popularity among designers of GGP agents. Providing GGP agents with the knowledge relevant to the game at hand in real time is, however, a challenging task. In this paper we propose two enhancements for MCTS in the context of GGP, aimed at improving the effectiveness of the simulations in real time based on in-game statistical feedback. The first extension allows early termination of lengthy and uninformative simulations while the second improves the action-selection strategy when both explored and unexplored actions are available. The methods are empirically evaluated in a state-of-the-art GGP agent and shown to yield an overall significant improvement in playing strength.

Deep Q-Network based real-time active disturbance rejection controller parameter tuning for multi-area interconnected power systems

For the problem of load frequency control (LFC) in multi-area interconnected power systems, a method of linear active disturbance rejection control (LADRC) that uses Deep Q-Network (DQN) to obtain controller parameters in real-time is designed. The main idea of combining the DQN and the LADRC is to equate the state and action in the environment of Reinforcement Learning (RL) to the output of the power system and the parameters of the controller, which will make the controller have learning ability and better adaptability. In order to achieve a better training effect, some improvements have been made to the action selection strategy and reward function. The proposed method is applied to the three-area and four-area interconnected power systems under the influence of disturbances, and the simulation results show the effectiveness of the proposed method. Compared with Proportional-Integral-Derivative (PID) controller and LADRC with fixed parameters, the method proposed in this paper has a better response effect in terms of overshoot and settling time of the power system.