Reinforcement Learning Research Articles

Research on trajectory planning based on deep reinforcement learning

Abstract For trajectory planning in a three-dimensional continuous space with multiple complex constraints, traditional optimal control methods face significant challenges, such as long computation times and poor dynamic performance. Using deep reinforcement learning algorithms for training agents in flight trajectory planning can effectively improve the dynamic performance of the planning algorithms. This paper focuses on training agents for UAV emergency landing and obstacle avoidance trajectory planning based on deep reinforcement learning algorithms. Within the reinforcement learning framework, we design the training environment, state space, action space, reward function, and neural network structure. To address common issues in reinforcement learning, such as poor generalization and slow training speed, we improve the training process using the concept of curriculum learning. Comparisons with pseudospectral methods demonstrate that deep reinforcement learning can effectively handle trajectory planning tasks in real flight environments, achieving trajectory quality close to that of pseudospectral methods while offering superior real-time performance.

The pupil outdoes the master: Imperfect demonstration-assisted trust region jamming policy optimization against frequency-hopping spread spectrum

Jamming decision-making is a pivotal component of modern electromagnetic warfare, wherein recent years have witnessed the extensive application of deep reinforcement learning techniques to enhance the autonomy and intelligence of wireless communication jamming decisions. However, existing researches heavily rely on manually designed customized jamming reward functions, leading to significant consumption of human and computational resources. To this end, under the premise of obviating designing task-customized reward functions, we propose a jamming policy optimization method that learns from imperfect demonstrations to effectively address the complex and high-dimensional jamming resource allocation problem against frequency hopping spread spectrum (FHSS) communication systems. To achieve this, a policy network is meticulously architected to consecutively ascertain jamming schemes for each jamming node, facilitating the construction of the dynamic transition within the Markov decision process. Subsequently, anchored in the dual-trust region concept, we design policy improvement and policy adversarial imitation phases. During the policy improvement phase, the trust region policy optimization method is utilized to refine the policy, while the policy adversarial imitation phase employs adversarial training to guide policy exploration using information embedded in demonstrations. Extensive simulation results indicate that our proposed method can approximate the optimal jamming performance trained under customized reward functions, even with rough binary reward settings, and also significantly surpass demonstration performance.

An innovative practical roadmap for optimal control strategies in malware propagation through the integration of RL with MPC

While there has been considerable research into optimal control formulations for mitigating cyber threats, a significant gap persists between the theoretical and numerical insights derived from such research and the practical implementation of these optimal mitigation strategies in real-time scenarios. This paper introduces a multifaceted approach to enhance and optimize optimal control strategies by seamlessly integrating reinforcement learning (RL) algorithms with model predictive control (MPC) techniques for the purpose of malware propagation control. Optimal control is a critical aspect of various domains, ranging from industrial processes and robotics to epidemiological modeling and cybersecurity. The traditional approaches to optimal control, particularly open-loop strategies, have limitations in adapting to dynamic and uncertain environments. This paper addresses these limitations by proposing a novel roadmap that leverages RL algorithms to fine-tune and adapt MPC parameters within the context of malware propagation containment. In sum, this practical roadmap is anticipated to serve as a valuable resource for researchers and practitioners engaged in the development of cybersecurity solutions.

Risk-averse supply chain management via robust reinforcement learning

Classical reinforcement learning (RL) may suffer performance degradation when the environment deviates from training conditions, limiting its application in risk-averse supply chain management. This work explores using robust RL in supply chain operations to hedge against environment inconsistencies and changes. Two robust RL algorithms, Qˆ-learning and β-pessimistic Q-learning, are examined against conventional Q-learning and a baseline order-up-to inventory policy. Furthermore, this work extends RL applications from forward to closed-loop supply chains. Two case studies are conducted using a supply chain simulator developed with agent-based modeling. The results show that Q-learning can outperform the baseline policy under normal conditions, but notably degrades under environment deviations. By comparison, the robust RL models tend to make more conservative inventory decisions to avoid large shortage penalties. Specifically, fine-tuned β-pessimistic Q-learning can achieve good performance under normal conditions and maintain robustness against moderate environment inconsistencies, making it suitable for risk-averse decision-making.

The docking control system of an autonomous underwater vehicle combining intelligent object recognition and deep reinforcement learning

This study develops a visual-based docking system (VDS) for an autonomous underwater vehicle (AUV), significantly enhancing docking performance by integrating intelligent object recognition and deep reinforcement learning (DRL). The system overcomes traditional navigation limitations in complex and unpredictable environments by using a variable information dock (VID) for precise multi-sensor docking recognition in the AUV. Employing image-based visual servoing (IBVS) technology, the VDS efficiently converts 2D visual data into accurate 3D motion control commands. It integrates the YOLO (short for You Only Look Once) algorithm for object recognition and the deep deterministic policy gradient (DDPG) algorithm, improving continuous motion control, docking accuracy, and adaptability. Experimental validation at the National Cheng Kung University towing tank demonstrates that the VDS enhances control stability and operational reliability, reducing the mean absolute error (MAE) in depth control by 42.03% and pitch control by 98.02% compared to the previous method. These results confirm the VDS's reliability and its potential for transforming AUV docking.

Quadrotor Trajectory Tracking using Combined Stochastic Model-Free Position and DDPG-based Attitude Control

This article presents a cascade controller for the quadrotor to track the desired trajectory effectively. Unlike previous approaches, this method avoids simplification and linearization assumptions, making it applicable in a wider range of scenarios. A novel linear quadratic tracking method is utilized, which takes into account both process noise and measurement noise while maintaining a model-free nature. Furthermore, the stability analysis of this stochastic method is thoroughly investigated. In terms of attitude control, a model-free approach is adopted. The Deep Deterministic Policy Gradient (DDPG) algorithm is implemented, leveraging an actor-critic network to handle the nonlinearities associated with attitude control. This model-free approach eliminates the need for an accurate model of the quadrotor's dynamics. Simulations are conducted to evaluate the performance of the proposed controller, and the results demonstrate its ability to effectively control the quadrotor, ensuring accurate trajectory tracking and stability.

GWO-ANFIS-RD3PG: A reinforcement learning approach with dynamic adjustment and dual replay mechanism for building energy forecasting

Reliable prediction of building energy consumption is essential for effective and sustainable energy management. Traditional forecasting methods, however, face challenges when dealing with sudden changes in energy consumption. To address the above-mentioned issue, we introduce the GWO-ANFIS-RD3PG prediction model in this paper. This model aims to ensure global forecasting accuracy while minimizing prediction errors at sudden change points. A Grey Wolf Optimization (GWO) algorithm with the Adaptive Neuro-Fuzzy Inference System (ANFIS) is proposed to predict the energy consumption type and its fluctuation magnitude for the next time step. Within the Recurrent Dual Experience Replay Buffer DDPG (RD3PG) prediction module, we employ Long Short-Term Memory (LSTM), as the actor network in the Deep Deterministic Policy Gradient (DDPG), to account for long-term dependencies in time series data. Notably, we introduce an adaptive action modification mechanism that allows the agent to optimize actions based on the output of the aforementioned ANFIS, enabling real-time adjustments during sudden change. To further enhance the model’s performance, we adopt a dual experience replay mechanism to store data samples from sudden change points, capturing insight information during these anomalies. Experimental results on two real-world datasets demonstrate that this method reduces MAE by 5.81% and 13.47%, and RMSE by 5.69% and 15.21% compared to the state-of-art models, showing the significance of the proposed method for energy efficiency improvement in real-world building operations.

Delay-aware resource allocation for partial computation offloading in mobile edge cloud computing

Mobile Edge Cloud Computing (MECC), as a promising partial computing offloading solution, has provided new possibilities for compute-intensive and delay-sensitive mobile applications, which can simultaneously leverage edge computing and cloud services. However, designing resource allocation strategies for MECC faces an extremely challenging problem of simultaneously satisfying the end-to-end latency requirements and minimum resource allocation of multiple mobile applications. To address this issue, we comprehensively consider the randomness of computing request arrivals, service time, and dynamic computing resources. We model the MECC network as a two-level tandem queue consisting of two sequential computing processing queues, each with multiple servers. We apply a deep reinforcement learning algorithm called Deep Deterministic Policy Gradient (DDPG) to learn the computing speed adjustment strategy for the tandem queue. This strategy ensures the end-to-end latency requirements of multiple mobile applications while preventing overuse of the total computing resources of edge servers and cloud servers. Numerous simulation experiments demonstrate that our approach is significantly superior to other methods in dynamic network environments.

Power flow distribution identification via machine learning algorithm and distributed resource clearing method

Abstract Under the background of dual carbon, a large number of distributed new energy sources are connected to the demand side, which leads to great risks in the operation of the power system. As an effective solution, distributed power transaction still has problems, such as high clearing time, unknown low-voltage distribution network structure, and resource scheduling ignoring user needs. Therefore, this paper proposes an optimal regulation method of demand-side resources based on distributed transactions. Based on the historical operation data of the distribution network, the eXtreme Gradient Boosting (XGBoost) algorithm is used to study the relationship between node injection power, node voltage, and node-side line power flow. It is also used to check the results of distributed transactions and design a market mechanism for rapid clearing and settlement of distributed transactions. The results of an example show that the identification of power flow distribution in a distribution network based on reinforcement learning can meet the security check requirements of distributed transaction results.

Online Q-learning for stochastic linear systems with state and control dependent noise

For continuous-time (CT) systems that are characterized by stochastic differential equations (SDEs) with completely unknown dynamics parameters, a reinforcement learning (RL)-based optimal control framework is presented in this paper. To obtain the near-optimal control policy, an online Q-learning algorithm is proposed by learning the data sampled from the system state trajectory, while an integral reinforcement learning (IRL) approach is developed in stochastic situation so as to formulate the Q-learning iterative algorithm. In particular, an actor/critic neural network (NN) structure is applied in iteration, where a critic approximator is used for estimating the designed Q-function while an actor approximator is for estimating the optimal control policy online. To ensure the convergence of iteration, the tuning laws of two neural networks are designed, respectively, by a gradient descent scheme. Moreover, the mean-square stability of the closed-loop system is proved through rigorous analysis, and the convergence to optimal solution is guaranteed as well.

Energy performance evaluation of the ASHRAE Guideline 36 control and reinforcement learning–based control using field measurements

This study evaluates the energy performance of ASHRAE Guideline 36–compliant control (ASHRAE 36 control) and reinforcement learning (RL)–based control through experimental field tests and a simulation study. Three field tests were conducted at Oak Ridge National Laboratory’s commercial building test facility in Oak Ridge, Tennessee: a baseline with a baseline conventional control, a test with ASHRAE 36 control, and a test with RL-based control. The selected ASHRAE 36 controls were trim and respond control, as well as variable air volume (VAV) box control. We compared the measured supply air temperature of the rooftop unit, VAV box supply air temperature, and VAV box supply airflow rate across the three test cases. The field data indicated that ASHRAE 36 controls operated as specified by ASHRAE Guideline 36. Based on these data, ASHRAE 36 control achieved a 45 % reduction in hourly averaged HVAC energy consumption compared with the baseline, and RL-based control achieved a 66 % reduction. These potential annual energy savings were confirmed using a calibrated whole-building energy model. Compared with the baseline, ASHRAE 36 control reduced HVAC energy consumption by 42 %, and RL-based control achieved a 54 % reduction. Furthermore, RL-based control reduced total HVAC energy consumption by 21 % more than ASHRAE 36 control.

Reinforcement learning and rational expectations equilibrium in limit order markets

This paper shows that simple payoff-based reinforcement learning can help to achieve rational expectations equilibrium in limit order markets. In equilibrium, speculators mainly supply liquidity, while liquidity consumption increases in the private values of no-speculators with intrinsic motives for trade. Driven by information acquisition of the non-speculators, liquidity consumption is hump-shaped in fundamental volatility for the speculators but U-shaped for the non-speculators. In contrast, liquidity supply decreases in fundamental volatility for the speculators but is hump-shaped for the non-speculators. Unlike the informed traders who trade on asset fundamentals, the uninformed traders trade more on order book and trading information.

Page Type-Aware Full-Sequence Program Scheduling via Reinforcement Learning in High Density SSDs

Page Type-Aware Full-Sequence Program Scheduling via Reinforcement Learning in High Density SSDs

Decision-making optimization for post-disaster restoration of multimodal transport networks in terms of resilience

IntroductionPublic health emergencies can have a ripple effect on the resilience of multimodal transport network, which will lead to problems such as route disruptions or blockages, route selection change and information transmission delay spreading to the whole network, further hindering the transportation planning and operational efficiency of the network. MethodsThis study constructs a multimodal transport route optimization model under uncertainty with the objective of the sum of transportation cost, transshipment cost, penalty cost and carbon emission cost. To enhance the computational efficiency of the model, a novel invasive weed optimization with memory and encoding value clustering capabilities is proposed. In addition, by fusing the Q-learning algorithm in reinforcement learning with the novel invasive weed algorithm, the action-value function table obtained from the training facilitates the selection of optimal routes. Based on empirical data, explore the sensitivity analysis of node disruptions, time windows, and fuzzy demand on route decision-making under public health emergencies. ResultsThe transport network is affected by public health emergencies, which makes the optimal route deviate from the expected goal, resulting in an increase in the total cost. The proportion of total cost is determined by the position of nodes in the network, with critical nodes suffering more losses than ordinary nodes. Reasonable setting of time windows and fuzzy demand intervals is an effective way to improve the resilience and transportation efficiency of multimodal transport network. ConclusionsThis study provides more applicable decision-making references for enterprises to prevent the risk of supply chain disruptions caused by public health emergencies.

Overcoming Delayed Feedback in Reinforcement Learning Using Actor Ensembles

Overcoming Delayed Feedback in Reinforcement Learning Using Actor Ensembles

Decrease in decision noise from adolescence into adulthood mediates an increase in more sophisticated choice behaviors and performance gain.

Learning and decision-making undergo substantial developmental changes, with adolescence being a particular vulnerable window of opportunity. In adolescents, developmental changes in specific choice behaviors have been observed (e.g., goal-directed behavior, motivational influences over choice). Elevated levels of decision noise, i.e., choosing suboptimal options, were reported consistently in adolescents. However, it remains unknown whether these observations, the development of specific and more sophisticated choice processes and higher decision noise, are independent or related. It is conceivable, but has not yet been investigated, that the development of specific choice processes might be impacted by age-dependent changes in decision noise. To answer this, we examined 93 participants (12 to 42 years) who completed 3 reinforcement learning (RL) tasks: a motivational Go/NoGo task assessing motivational influences over choices, a reversal learning task capturing adaptive decision-making in response to environmental changes, and a sequential choice task measuring goal-directed behavior. This allowed testing of (1) cross-task generalization of computational parameters focusing on decision noise; and (2) assessment of mediation effects of noise on specific choice behaviors. Firstly, we found only noise levels to be strongly correlated across RL tasks. Second, and critically, noise levels mediated age-dependent increases in more sophisticated choice behaviors and performance gain. Our findings provide novel insights into the computational processes underlying developmental changes in decision-making: namely a vital role of seemingly unspecific changes in noise in the specific development of more complex choice components. Studying the neurocomputational mechanisms of how varying levels of noise impact distinct aspects of learning and decision processes may also be key to better understand the developmental onset of psychiatric diseases.

Deep reinforcement learning for autonomous SideLink radio resource management in platoon-based C-V2X networks: An overview

Dynamic and autonomous SideLink (SL) Radio Resource Management (RRM) is essential for platoon-based cellular vehicular networks. However, this task is challenging due to several factors. These include the limited spectrum below 6 GHz, stringent vehicle-to-everything (V2X) communications requirements, uncertain and dynamic environments, limited vehicle sensing capabilities, and inherent distributed operation. These limitations often lead to resource collisions, data packet loss, and increased latency. Current standardized approaches in Long-Term Evolution-V2X (LTE-V2X) and New Radio-V2X (NR-V2X) rely on random resource selection, limiting their efficiency. Moreover, RRM is inherently a complex combinatorial optimization problem. It may involve conflicting objectives and constraints, making traditional approaches inadequate. Platoon-based communication necessitates careful resource allocation to support a diverse mix of communication types. These include safety-critical control messaging within platoons, less time-sensitive traffic management information between platoons, and even infotainment services like media streaming. Optimizing resource sharing inter- and intra- platoons is crucial to avoid excessive interference and ensure overall network performance. Deep Reinforcement Learning (DRL), combining Deep Learning (DL) and Reinforcement Learning (RL), has recently been investigated for network resource management. It offers a potential solution for these challenges. A DRL agent, represented by deep neural networks, interacts with the environment and learns optimal decision-making through trial and error. This paper overviews proposed DRL-based methods for autonomous SL RRM in single and multi-agent platoon-based C-V2X networks. It considers both intra- and inter-platoon communications with their specific requirements. We discuss the components of Markov Decision Processes (MDP) used to model the sequential decision-making of RRM. We then detail the DRL algorithms, training paradigms, and insights on the achieved results. Finally, we highlight challenges in existing works and suggest strategies for addressing them.

Comparison of Empirical and Reinforcement Learning (RL)-Based Control Based on Proximal Policy Optimization (PPO) for Walking Assistance: Does AI Always Win?

Comparison of Empirical and Reinforcement Learning (RL)-Based Control Based on Proximal Policy Optimization (PPO) for Walking Assistance: Does AI Always Win?

Semi-Supervised Intrusion Detection System for In-Vehicle Networks Based on Variational Autoencoder and Adversarial Reinforcement Learning

Despite the affordability, simplicity, and efficiency of controller area network (CAN) protocols, the security vulnerability remains a major challenge. Currently, a machine learning-based intrusion detection system (IDS) is considered an effective approach for improving security in CAN by identifying malicious attacks. However, earlier studies that relied on supervised learning methods required considerable amounts of labeled data. Data collection from vehicles is time-consuming and expensive. Furthermore, the obtained data exhibited a class imbalance, which presents further challenges in the analysis and model training. Thus, we propose a semi-supervised learning-based IDS that combines variational autoencoder (VAE) and adversarial reinforcement learning for the multi-class classification of both known and unknown attacks. The proposed system capitalizes on the diverse patterns inherent in unlabeled data, transforming this data space into one that is more conducive to classification. Concurrently, adversarial agents in the reinforcement learning algorithm interact competitively, progressively enhancing their ability to intelligently classify and select samples. To reduce the reliance on labeled data and effectively exploit them, we utilize a pseudo-labeling process for pre-training. Experimental results indicate that the proposed model achieves more effective classification while requiring less labeled data compared to other baseline models for known attacks. By inheriting the advantages of VAE, promising results demonstrate that the proposed system detects unknown attacks containing similar or completely different characteristics with high F1 scores exceeding 0.9 and 0.84, respectively. Finally, the proposed system was demonstrated to be a lightweight model for the expeditious detection of malevolent messages introduced into in-vehicle networks to ensure minimal latency.

A Model-Based Reinforcement Learning Algorithm for Multi-Agent Cooperation Nash Equilibrium With Unstable Communication

Solving Nash equilibrium is important to multi-agent systems, in which the communication is an important factor. However, many proposed reinforcement learning (RL) based algorithms take into account the communication factors by assuming stable communication conditions, which does not hold in the real environment. In this paper, we analyze the effect of a typical RL algorithm in the cases of unstable communication and communication failure, which causes information loss between agents, leading to isolation of agents and affecting algorithm convergence. Then, we propose a model-based RL algorithm to solve Nash equilibrium for multi-agent systems when agents are isolated, and prove its convergence and rationality through mathematical proofs. The simulations results show the effectiveness of the proposed algorithm.