Reinforcement Learning Algorithm Research Articles

Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search

In the past few years, Differentiable Neural Architecture Search (DNAS) rapidly imposed itself as the trending approach to automate the discovery of deep neural network architectures. This rise is mainly due to the popularity of DARTS (Differentiable ARchitecTure Search), one of the first major DNAS methods. In contrast with previous works based on Reinforcement Learning or Evolutionary Algorithms, DNAS is faster by several orders of magnitude and uses fewer computational resources. In this comprehensive survey, we focused specifically on DNAS and reviewed recent approaches in this field. Furthermore, we proposed a novel challenge-based taxonomy to classify DNAS methods. We also discussed the contributions brought to DNAS in the past few years and its impact on the global NAS field. Finally, we concluded by giving some insights into future research directions for the DNAS field.

Beyond Static Obstacles: Integrating Kalman Filter with Reinforcement Learning for Drone Navigation

Autonomous drones offer immense potential in dynamic environments, but their navigation systems often struggle with moving obstacles. This paper presents a novel approach for drone trajectory planning in such scenarios, combining the Interactive Multiple Model (IMM) Kalman filter with Proximal Policy Optimization (PPO) reinforcement learning (RL). The IMM Kalman filter addresses state estimation challenges by modeling the potential motion patterns of moving objects. This enables accurate prediction of future object positions, even in uncertain environments. The PPO reinforcement learning algorithm then leverages these predictions to optimize the drone’s real-time trajectory. Additionally, the capability of PPO to work with continuous action spaces makes it ideal for the smooth control adjustments required for safe navigation. Our simulation results demonstrate the effectiveness of this combined approach. The drone successfully navigates complex dynamic environments, achieving collision avoidance and goal-oriented behavior. This work highlights the potential of integrating advanced state estimation and reinforcement learning techniques to enhance autonomous drone capabilities in unpredictable settings.

Quadratic Tracking Control of Linear Stochastic Systems with Unknown Dynamics Using Average Off-Policy Q-Learning Method

This article investigates the optimal tracking control problem for data-based stochastic discrete-time linear systems. An average off-policy Q-learning algorithm is proposed to solve the optimal control problem with random disturbances. Compared with the existing off-policy reinforcement learning (RL) algorithm, the proposed average off-policy Q-learning algorithm avoids the assumption of an initial stability control. First, a pole placement strategy is used to design an initial stable control for systems with unknown dynamics. Second, the initial stable control is used to design a data-based average off-policy Q-learning algorithm. Then, this algorithm is used to solve the stochastic linear quadratic tracking (LQT) problem, and a convergence proof of the algorithm is provided. Finally, numerical examples show that this algorithm outperforms other algorithms in a simulation.

Reconfigurable-Intelligent-Surface-Enhanced Dynamic Resource Allocation for the Social Internet of Electric Vehicle Charging Networks with Causal-Structure-Based Reinforcement Learning

Charging stations and electric vehicle (EV) charging networks signify a significant advancement in technology as a frontier application of the Social Internet of Things (SIoT), presenting both challenges and opportunities for current 6G wireless networks. One primary challenge in this integration is limited wireless network resources, particularly when serving a large number of users within distributed EV charging networks in the SIoT. Factors such as congestion during EV travel, varying EV user preferences, and uncertainties in decision-making regarding charging station resources significantly impact system operation and network resource allocation. To address these challenges, this paper develops a novel framework harnessing the potential of emerging technologies, specifically reconfigurable intelligent surfaces (RISs) and causal-structure-enhanced asynchronous advantage actor–critic (A3C) reinforcement learning techniques. This framework aims to optimize resource allocation, thereby enhancing communication support within EV charging networks. Through the integration of RIS technology, which enables control over electromagnetic waves, and the application of causal reinforcement learning algorithms, the framework dynamically adjusts resource allocation strategies to accommodate evolving conditions in EV charging networks. An essential aspect of this framework is its ability to simultaneously meet real-world social requirements, such as ensuring efficient utilization of network resources. Numerical simulation results validate the effectiveness and adaptability of this approach in improving wireless network efficiency and enhancing user experience within the SIoT context. Through these simulations, it becomes evident that the developed framework offers promising solutions to the challenges posed by integrating the SIoT with EV charging networks.

Control of Qubit Dynamics Using Reinforcement Learning

The progress in machine learning during the last decade has had a considerable impact on many areas of science and technology, including quantum technology. This work explores the application of reinforcement learning (RL) methods to the quantum control problem of state transfer in a single qubit. The goal is to create an RL agent that learns an optimal policy and thus discovers optimal pulses to control the qubit. The most crucial step is to mathematically formulate the problem of interest as a Markov decision process (MDP). This enables the use of RL algorithms to solve the quantum control problem. Deep learning and the use of deep neural networks provide the freedom to employ continuous action and state spaces, offering the expressivity and generalization of the process. This flexibility helps to formulate the quantum state transfer problem as an MDP in several different ways. All the developed methodologies are applied to the fundamental problem of population inversion in a qubit. In most cases, the derived optimal pulses achieve fidelity equal to or higher than 0.9999, as required by quantum computing applications. The present methods can be easily extended to quantum systems with more energy levels and may be used for the efficient control of collections of qubits and to counteract the effect of noise, which are important topics for quantum sensing applications.

Multi-agent cooperative optimal scheduling strategy of integrated energy system in urban area under extreme events

Because the global climate change intensifies as well as the natural disasters frequently occur, extreme events have caused serious impacts on the energy system in urban areas, and at the same time, they have brought great challenges to the supply and scheduling of urban energy systems. Therefore, in order to better integrate and manage various energy resources in urban areas, a Deep Q-Leaning Network-Quasi Upper Confidence Bound model is innovatively constructed using deep reinforcement learning technology to learn the state and behavior mapping relationship of energy system. Use deep learning to fit complex nonlinear models to optimize the entire energy system. Compare and verify the experiment with the real energy system. The improved Deep reinforcement learning algorithm is compared with Q-learning model, PDWoLF PHC algorithm model, Quasi Upper Confidence Bound algorithm model and deep Q-Leaning Network algorithm model. The results show that the research algorithm has the smallest instantaneous error value and absolute value of frequency deviation for area control, and the average value of the research algorithm in the absolute value of the frequency deviation is reduced by 45%–73% compared to other algorithms; over time, the unit output power of the research algorithm is able to flexibly track the stochastic square wave loads. Therefore, the proposed system strategies can provide feasible solutions to meet the challenges of extreme events and promote the sustainable development and safe operation of urban energy systems.

Bio-Inspired Intelligent Swarm Confrontation Algorithm for a Complex Urban Scenario

This paper considers the confrontation problem for two tank swarms of equal size and capability in a complex urban scenario. Based on the Unity platform (2022.3.20f1c1), the confrontation scenario is constructed featuring multiple crossing roads. Through the analysis of a substantial amount of biological data and wildlife videos regarding animal behavioral strategies during confrontations for hunting or food competition, two strategies are been utilized to design a novel bio-inspired intelligent swarm confrontation algorithm. The first one is the “fire concentration” strategy, which assigns a target for each tank in a way that the isolated opponent will be preferentially attacked with concentrated firepower. The second one is the “back and forth maneuver” strategy, which makes the tank tactically retreat after firing in order to avoid being hit when the shell is reloading. Two state-of-the-art swarm confrontation algorithms, namely the reinforcement learning algorithm and the assign nearest algorithm, are chosen as the opponents for the bio-inspired swarm confrontation algorithm proposed in this paper. Data of comprehensive confrontation tests show that the bio-inspired swarm confrontation algorithm has significant advantages over its opponents from the aspects of both win rate and efficiency. Moreover, we discuss how vital algorithm parameters would influence the performance indices.

The well-worn route revisited: Striatal and hippocampal system contributions to familiar route navigation.

Classic research has shown a division in the neuroanatomical structures that support flexible (e.g., short-cutting) and habitual (e.g., familiar route following) navigational behavior, with hippocampal-caudate systems associated with the former and putamen systems with the latter. There is, however, disagreement about whether the neural structures involved in navigation process particular forms of spatial information, such as associations between constellations of cues forming a cognitive map, versus single landmark-action associations, or alternatively, perform particular reinforcement learning algorithms that allow the use of different spatial strategies, so-called model-based (flexible) or model-free (habitual) forms of learning. We sought to test these theories by asking participants (N = 24) to navigate within a virtual environment through a previously learned, 9-junction route with distinctive landmarks at each junction while undergoing functional magnetic resonance imaging (fMRI). In a series of probe trials, we distinguished knowledge of individual landmark-action associations along the route versus knowledge of the correct sequence of landmark-action associations, either by having absent landmarks, or "out-of-sequence" landmarks. Under a map-based perspective, sequence knowledge would not require hippocampal systems, because there are no constellations of cues available for cognitive map formation. Within a learning-based model, however, responding based on knowledge of sequence would require hippocampal systems because prior context has to be utilized. We found that hippocampal-caudate systems were more active in probes requiring sequence knowledge, supporting the learning-based model. However, we also found greater putamen activation in probes where navigation based purely on sequence memory could be planned, supporting models of putamen function that emphasize its role in action sequencing.

Energy-saving design of underground metro vertical alignment with deep reinforcement learning

The design of metro vertical alignment greatly affects train traction energy consumption. This study develops a deep reinforcement learning (DRL) algorithm for an energy-saving design of underground metro vertical alignment and analyzes the influence of train traction energy consumption calculation factors (CFs) on energy-saving design. First, a metro train operation simulation model is adopted using the approximate integral method, and the value range of CFs as input parameters is determined. Second, the 'agent', 'state', 'action', and 'reward' of the metro vertical alignment environment and the corresponding alignment constraints are determined. Third, a Dueling Double Deep Q Network (D3QN) is constructed to design the energy-saving metro vertical alignment. The proposed model is applied to a real-world metro case study, and the influence of CFs (design speed, limiting gradient in reverse slopes, and passenger capacity) on energy-saving design is analyzed. Finally, compared to manual design, the D3QN-based design outperforms GA-based design and DQN-based design, and demonstrates a remarkable energy savings of 6.17%.

Multi-objective reinforcement learning based on nonlinear scalarization and long-short-term optimization

Purpose Many practical control problems require achieving multiple objectives, and these objectives often conflict with each other. The existing multi-objective evolutionary reinforcement learning algorithms cannot achieve good search results when solving such problems. It is necessary to design a new multi-objective evolutionary reinforcement learning algorithm with a stronger searchability. Design/methodology/approach The multi-objective reinforcement learning algorithm proposed in this paper is based on the evolutionary computation framework. In each generation, this study uses the long-short-term selection method to select parent policies. The long-term selection is based on the improvement of policy along the predefined optimization direction in the previous generation. The short-term selection uses a prediction model to predict the optimization direction that may have the greatest improvement on overall population performance. In the evolutionary stage, the penalty-based nonlinear scalarization method is used to scalarize the multi-dimensional advantage functions, and the nonlinear multi-objective policy gradient is designed to optimize the parent policies along the predefined directions. Findings The penalty-based nonlinear scalarization method can force policies to improve along the predefined optimization directions. The long-short-term optimization method can alleviate the exploration-exploitation problem, enabling the algorithm to explore unknown regions while ensuring that potential policies are fully optimized. The combination of these designs can effectively improve the performance of the final population. Originality/value A multi-objective evolutionary reinforcement learning algorithm with stronger searchability has been proposed. This algorithm can find a Pareto policy set with better convergence, diversity and density.

Design and Development of Multi-Agent Reinforcement Learning Intelligence on the Robotarium Platform for Embedded System Applications

This research explores the use of Q-Learning for real-time swarm (Q-RTS) multi-agent reinforcement learning (MARL) algorithm for robotic applications. This study investigates the efficacy of Q-RTS in the reducing convergence time to a satisfactory movement policy through the successful implementation of four and eight trained agents. Q-RTS has been shown to significantly reduce search time in terms of training iterations, from almost a million iterations with one agent to 650,000 iterations with four agents and 500,000 iterations with eight agents. The scalability of the algorithm was addressed by testing it on several agents’ configurations. A central focus was placed on the design of a sophisticated reward function, considering various postures of the agents and their critical role in optimizing the Q-learning algorithm. Additionally, this study delved into the robustness of trained agents, revealing their ability to adapt to dynamic environmental changes. The findings have broad implications for improving the efficiency and adaptability of robotic systems in various applications such as IoT and embedded systems. The algorithm was tested and implemented using the Georgia Tech Robotarium platform, showing its feasibility for the above-mentioned applications.

A DRL-based Partial Charging Algorithm for Wireless Rechargeable Sensor Networks

Breakthroughs in Wireless Energy Transfer (WET) technologies have revitalized Wireless Rechargeable Sensor Networks (WRSNs). However, how to schedule mobile chargers rationally has been quite a tricky problem. Most of the current work does not consider the variability of scenarios and how many mobile chargers should be scheduled as the most appropriate for each dispatch. At the same time, the focus of most work on the mobile charger scheduling problem has always been on reducing the number of dead nodes, and the most critical metric of network performance, packet arrival rate, is relatively neglected. In this paper, we develop a DRL-based Partial Charging (DPC) algorithm. Based on the number and urgency of charging requests, we classify charging requests into four scenarios. And for each scenario, we design a corresponding request allocation algorithm. Then, a Deep Reinforcement Learning (DRL) algorithm is employed to train a decision model using environmental information to select which request allocation algorithm is optimal for the current scenario. After the allocation of charging requests is confirmed, to improve the Quality of Service (QoS), i.e., the packet arrival rate of the entire network, a partial charging scheduling algorithm is designed to maximize the total charging duration of nodes in the ideal state while ensuring that all charging requests are completed. In addition, we analyze the traffic information of the nodes and use the Analytic Hierarchy Process (AHP) to determine the importance of the nodes to compensate for the inaccurate estimation of the node’s remaining lifetime in realistic scenarios. Simulation results show that our proposed algorithm outperforms the existing algorithms regarding the number of alive nodes and packet arrival rate.

Fault-Tolerant Control for Multi-UAV Exploration System via Reinforcement Learning Algorithm

In the UAV swarm, the degradation in the health status of some UAVs often brings negative effects to the system. To compensate for the negative effect, we present a fault-tolerant Multi-Agent Reinforcement Learning Algorithm that can control an unstable Multiple Unmanned Aerial Vehicle (Multi-UAV) system to perform exploration tasks. Different from traditional multi-agent methods that require the agents to remain healthy during task execution, our approach breaks this limitation and allows the agents to change status during the task. In our algorithm, the agent can accept both the adjacency state matrix about the neighboring agents and a kind of healthy status vector to integrate both and generate the communication topology. During this process, the agents with poor health status are given more attention for returning to normal status. In addition, we integrate a temporal convolution module into our algorithm and enable the agent to capture the temporal information during the task. We introduce a scenario regarding Multi-UAV ground exploration, where the health status of UAVs gradually weakens over time before dropping into a fault status; the UAVs require rescues from time to time. We conduct some experiments in this scenario and verify our algorithm. Our algorithm can increase the drone’s survival rate and make the swarm perform better.

Reinforcement Learning for Closed-loop Regulation of Cardiovascular System with Vagus Nerve Stimulation: A Computational Study.

(Objective). Vagus nerve stimulation (VNS) is being investigated as a potential therapy for cardiovascular diseases including heart failure, cardiac arrhythmia, and hypertension. The lack of a systematic approach for controlling and tuning the VNS parameters poses a significant challenge. Closed-loop VNS strategies combined with artificial intelligence (AI) approaches offer a framework for systematically learning and adapting the optimal stimulation parameters. In this study, we presented an interactive AI framework using reinforcement learning (RL) for automated data-driven design of closed-loop VNS control systems in a computational study. (Approach). Multiple simulation environments with a standard application programming interface were developed to facilitate the design and evaluation of the automated data-driven closed-loop VNS control systems. These environments simulate the hemodynamic response to multi-location VNS using biophysics-based computational models of healthy and hypertensive rat cardiovascular systems in resting and exercise states. We designed and implemented the RL-based closed-loop VNS control frameworks in the context of controlling the heart rate (HR) and the mean arterial pressure (MAP) for a set point tracking task. Our experimental design included two approaches; a general policy using deep RL algorithms and a sample-efficient adaptive policy using probabilistic inference for learning and control (PILCO). (Main results). Our simulation results demonstrated the capabilities of the closed-loop RL-based approaches to learn optimal VNS control policies and to adapt to variations in the target set points and the underlying dynamics of the cardiovascular system. Our findings highlighted the trade-off between sample-efficiency and generalizability, providing insights for proper algorithm selection. Finally, we demonstrated that transfer learning improves the sample efficiency of Deep RL algorithms allowing the development of more efficient and personalized closed-loop VNS systems. (Significance). We demonstrated the capability of RL-based closed-loop VNS systems. Our approach provided a systematic adaptable framework for learning control strategies without requiring prior knowledge about the underlying dynamics.

Adaptive Critic Learning-Based Optimal Bipartite Consensus for Multiagent Systems With Prescribed Performance.

Developing a distributed bipartite optimal consensus scheme while ensuring user-predefined performance is essential in practical applications. Existing approaches to this problem typically require a complex controller structure due to adopting an identifier-actor-critic framework and prescribed performance cannot be guaranteed. In this work, an adaptive critic learning (ACL)-based optimal bipartite consensus scheme is developed to bridge the gap. A newly designed error scaling function, which defines the user-predefined settling time and steady accuracy without relying on the initial conditions, is then integrated into a cost function. The backstepping framework combines the ACL and integral reinforcement learning (IRL) algorithm to develop the adaptive optimal bipartite consensus scheme, which contributes a critic-only controller structure by removing the identifier and actor networks in the existing methods. The adaptive law of the critic network is derived by the gradient descent algorithm and experience replay to minimize the IRL-based residual error. It is shown that a compute-saving learning mechanism can achieve the optimal consensus, and the error variables of the closed-loop system are uniformly ultimately bounded (UUB). Besides, in any bounded initial condition, the evolution of bipartite consensus is limited to a user-prescribed boundary under bounded initial conditions. The illustrative simulation results validate the efficacy of the approach.

DePAint: a decentralized safe multi-agent reinforcement learning algorithm considering peak and average constraints

DePAint: a decentralized safe multi-agent reinforcement learning algorithm considering peak and average constraints

An adversarial environment reinforcement learning-driven intrusion detection algorithm for Internet of Things

The increasing prevalence of Internet of Things (IoT) systems has made them attractive targets for malicious actors. To address the evolving threats and the growing complexity of detection, there is a critical need to search for and develop new algorithms that are fast and robust in detecting and classifying dangerous network traffic. In this context, deep reinforcement learning (DRL) is gaining recognition as a prospective solution in numerous fields as it enables autonomous agents to cooperate with their environment for decision-making without relying on human experts. This article presents an innovative approach to intrusion detection in IoT systems using an adversarial reinforcement learning (RL) algorithm known for its exceptional predictive capabilities. The predictive process relies on a classifier, implemented as a streamlined and highly efficient neural network. Embedded within this classifier is a policy function meticulously trained using an innovative RL model. Importantly, this model ensures that the environment’s behavior is dynamically fine-tuned simultaneously with the learning process, improving the overall effectiveness of the intrusion detection approach. The efficiency of our proposal was assessed using the Bot-IoT database, consisting of a mixture of legitimate IoT network traffic and simulated attack scenarios. Our scheme shows superior performance compared to existing ones. Therefore, our approach to IoT intrusion detection can be considered a valuable alternative to existing methods, capable of significantly improving the IoT systems’ security.

Optimizing Fuel Injection Timing for Multiple Injection Using Reinforcement Learning and Functional Mock-up Unit for a Small-bore Diesel Engine

<div>Reinforcement learning (RL) is a computational approach to understanding and automating goal-directed learning and decision-making. The difference from other computational approaches is the emphasis on learning by an agent from direct interaction with its environment to achieve long-term goals [<span>1</span>]. In this work, the RL algorithm was implemented using Python. This then enables the RL algorithm to make decisions to optimize the output from the system and provide real-time adaptation to changes and their retention for future usage. A diesel engine is a complex system where a RL algorithm can address the NO<sub>x</sub>–soot emissions trade-off by controlling fuel injection quantity and timing. This study used RL to optimize the fuel injection timing to get a better NO–soot trade-off for a common rail diesel engine. The diesel engine utilizes a pilot–main and a pilot–main–post-fuel injection strategy. Change of fuel injection quantity was not attempted in this study as the main objective was to demonstrate the use of RL algorithms while maintaining a constant indicated mean effective pressure. A change in fuel quantity has a larger influence on the indicated mean effective pressure than a change in fuel injection timing. The focus of this work was to present a novel methodology of using the 3D combustion data from analysis software in the form of a functional mock-up unit (FMU) and showcasing the implementation of a RL algorithm in Python language to interact with the FMU to reduce the NO and soot emissions by suggesting changes to the main injection timing in a pilot–main and pilot–main–post-injection strategy. RL algorithms identified the operating injection strategy, i.e., main injection timing for a pilot–main and pilot–main–post-injection strategy, reducing NO emissions from 38% to 56% and soot emissions from 10% to 90% for a range of fuel injection strategies.</div>

Impact of memory on beamforming optimization for DDPG-assisted RIS-multi-user system

Recently, reconfigurable intelligent surfaces (RIS) have garnered considerable attention as indispensable components driving the evolution of future 6G wireless communication systems. This heightened interest is primarily attributed to notable advancements in programmable meta-material fabrication, which enable the creation of highly versatile and adaptable surfaces. These surfaces, often referred to as intelligent reflecting arrays, represent a significant departure from the conventional capabilities of massive multiple-input multiple-output (MIMO) systems, thereby catalyzing the emergence of intelligent radio environments characterized by enhanced flexibility and efficiency. Our study is dedicated to the comprehensive exploration of coordinated design strategies that encompass both the transmission beamforming matrix at the base station and the phase shift matrix at the RIS. Leveraging recent breakthroughs in deep reinforcement learning (DRL), our approach harnesses the power of a Long Short-Term Memory (LSTM) based DRL algorithm. This algorithm orchestrates the joint design process through iterative interactions with the environment, strategically guided by predefined rewards operating within a continuous framework of states and actions. While recent research has underscored the efficacy of LSTM-based architectures in bolstering the learning capacity of reinforcement learning (RL) algorithms and simplifying the search process, our investigation reveals a nuanced insight. Contrary to prevailing trends, we find that the integration of memory into the Deep Deterministic Policy Gradient (DDPG) with LSTM (DDPG-LSTM) algorithm yields unexpected consequences, negatively impacting the system's overall performance. Through preliminary simulation results, we empirically demonstrate the adverse effect of memory on DDPG performance when applied to RIS systems. This finding not only sheds light on the complexities of optimizing 6G wireless communication systems but also underscores the importance of careful algorithmic design and parameter tuning in achieving desired outcomes in emerging technologies.

Deep reinforcement learning for artificial upwelling energy management

The potential of artificial upwelling in stimulating seaweed growth, consequently enhancing ocean carbon sequestration has been gaining increasing attention in recent years. This has led to the development of the first solar-powered and air-lifted artificial upwelling system (AUS) in China. However, effective scheduling of the air injection system and energy storage system in dynamic marine environments remains a crucial challenge in the operation of the AUS, as it holds the potential to significantly improve system performance. To tackle this challenge, we propose a novel energy management approach that utilizes the deep reinforcement learning (DRL) algorithm to determine the optimal operational parameters of the AUS at each time interval. Specifically, we formulate the energy optimization problem as a Markov decision process and integrate the quantile network in distributional reinforcement learning with the deep dueling network to solve the problem. Through extensive simulations, we evaluate the performance of our algorithm and demonstrate its superior effectiveness over traditional rule-based approaches and other DRL algorithms in enhancing energy utilization while ensuring the secure and reliable operation of the AUS. Our findings suggest that a DRL-based approach offers a promising way to provide valuable guides for the operation of the AUS and enhance the sustainability of seaweed cultivation and carbon sequestration in the ocean.