RL Algorithm Research Articles

AugmenToxic: Leveraging Reinforcement Learning to Optimize LLM Instruction Fine-Tuning for Data Augmentation to Enhance Toxicity Detection

Addressing the challenge of toxic language in online discussions is crucial for the development of effective toxicity detection models. This pioneering work focuses on addressing imbalanced datasets in toxicity detection by introducing a novel approach to augment toxic language data. We create a balanced dataset by instructing fine-tuning of Large Language Models (LLMs) using Reinforcement Learning with Human Feedback (RLHF). Recognizing the challenges in collecting sufficient toxic samples from social media platforms for building a balanced dataset, our methodology involves sentence-level text data augmentation through paraphrasing existing samples using optimized generative LLMs. Leveraging generative LLM, we utilize the Proximal Policy Optimizer (PPO) as the RL algorithm to fine-tune the model further and align it with human feedback. In other words, we start by fine-tuning a LLM using an instruction dataset, specifically tailored for the task of paraphrasing while maintaining semantic consistency. Next, we apply PPO and a reward function, to further fine-tune (optimize) the instruction-tuned LLM. This RL process guides the model in generating toxic responses. We utilize the Google Perspective API as a toxicity evaluator to assess generated responses and assign rewards/penalties accordingly. This approach guides LLMs through PPO and the reward function, transforming minority class samples into augmented versions. The primary goal of our methodology is to create a balanced and diverse dataset to enhance the accuracy and performance of classifiers in identifying instances from the minority class. Utilizing two publicly available toxic datasets, we compared various techniques with our proposed method for generating toxic samples, demonstrating that our approach outperforms all others in producing a higher number of toxic samples. Starting with an initial 16,225 toxic prompts, our method successfully generated 122,951 toxic samples with a toxicity score exceeding 30%. Subsequently, we developed various classifiers using the generated balanced datasets and applied a cost-sensitive learning approach to the original imbalanced dataset. The findings highlight the superior performance of classifiers trained on data generated using our proposed method. These results highlight the importance of employing RL and a data-agnostic model as a reward mechanism for augmenting toxic data, thereby enhancing the robustness of toxicity detection models.

Implementation of Hybrid Adaptive Learning Algorithm for Task Offloading

Objectives: Offloading tasks in edge computing (EC) plays an important role in optimizing resource utilization and enhancing the system performance. This paper studies various AI-based computation offloading (CO) strategies and proposes a hybrid Adaptive Learning Algorithm, that reduces the latency significantly compared to traditional RL-based on-policy and off-policy algorithms namely Q-Learning (QL) and State Action Reward State Action (SARSA) for CO in an EC environment. The paper evaluates and compares the efficiency of the proposed algorithms in optimizing dynamic offloading decisions, focusing on factors such as latency. Methods: The research is inspired by existing literature that has discovered the various applications of QL and SARSA in CO and mobile EC. This paper inspects how these algorithms perform in terms of reducing overall latency and proposes an adaptive hybrid algorithm. Novelty: Although other AI-based techniques exist in earlier research on CO in EC, the primary novelty proposed is the addition of a hybrid adaptive RL algorithm. Factors impacting the task offloading decision are crucial in the proposed algorithm. Moreover, the decision is not based only on static policies; the current state is the driving factor for decision-making. Findings: The exploration and exploitation handshake is very promising in reducing latency, as seen in the simulation results. The proposed algorithm outperforms Q-Learning and SARSA in terms of latency. Keywords: Computation offloading, Mobile Edge Computing, Reinforcement learning, Q­learning, SARSA

Learning to construct a solution for UAV path planning problem with positioning error correction

Unmanned aerial vehicles (UAVs) are advanced flight systems. However, their positioning systems cause distance-dependent errors during flight. This study seeks to solve the UAV path planning problem with positioning error correction (UPEC) with an end-to-end method. Traditional methods struggle to balance solution quality and computational overload, and often have limited utilisation of scenario information. To overcome these issues, we propose a path planning model (PPM) based on deep reinforcement learning to solve the UPEC. The model has a complete structure that includes a mathematical model, feature engineering, solution process, neural policy network, scenario generation, training process, and test solution mechanism. Specifically, we first establish a Markov decision process (MDP) for UPEC and apply feature engineering with effective features to support decision-making. We then introduce a path planning neural network (PPNN) to represent the MDP policy. Based on the dataset generated from the multi-rule combination validation, we train the PPNN using the proposed RL algorithm with storage pool. Furthermore, we propose a backtracking mechanism to guarantee solution feasibility during the construction process. Extensive experiments demonstrate that the proposed PPM outperforms existing state-of-the-art algorithms in terms of solution quality and timeliness, and the backtracking mechanism effectively improves the scenario completion rate. The model study indicates the efficacy of our training algorithm and the generalisation of the PPNN. Additionally, our construction process is problem-tailored and more suitable for addressing UPEC than iterative search algorithms, because it effectively mitigates the impact of invalid nodes.

Fish-inspired tracking of underwater turbulent plumes

Autonomous ocean-exploring vehicles have begun to take advantage of onboard sensor measurements of water properties such as salinity and temperature to locate oceanic features in real time. Such targeted sampling strategies enable more rapid study of ocean environments by actively steering towards areas of high scientific value. Inspired by the ability of aquatic animals to navigate via flow sensing, this work investigates hydrodynamic cues for accomplishing targeted sampling using a palm-sized robotic swimmer. As proof-of-concept analogy for tracking hydrothermal vent plumes in the ocean, the robot is tasked with locating the center of turbulent jet flows in a 13,000-liter water tank using data from onboard pressure sensors. To learn a navigation strategy, we first implemented RL on a simulated version of the robot navigating in proximity to turbulent jets. After training, the RL algorithm discovered an effective strategy for locating the jets by following transverse velocity gradients sensed by pressure sensors located on opposite sides of the robot. When implemented on the physical robot, this gradient following strategy enabled the robot to successfully locate the turbulent plumes at more than twice the rate of random searching. Additionally, we found that navigation performance improved as the distance between the pressure sensors increased, which can inform the design of distributed flow sensors in ocean robots. Our results demonstrate the effectiveness and limits of flow-based navigation for autonomously locating hydrodynamic features of interest.

An Alternative Reinforcement Learning (ARL) control strategy for data center air-cooled HVAC systems

Energy efficiency of data center is of great concern globally due to their large amount of energy consumption and the foreseeable growth in the demand of digital services in the future. Advanced control strategies are needed to reduce energy consumption. However, the optimization of data center HVAC control is a challenge task due to the complexity of the thermal dynamic models of buildings and uncertainties associated with both server loads and outdoor temperature. In this paper, we propose a new control strategy called Alternately-Reinforcement Learning-control(ARL), which realizes the alternating control of RL and proportional, integral and derivative (PID) for optimizing the control of air-cooled HVAC systems in data centers. The control object of the ARL is the speed set of the compressor and the condensing fan, and the control goal is to minimize energy consumption while maintaining temperature stability. The applied RL algorithm is Deep deterministic policy gradient (DDPG) and Proximal policy optimization (PPO). We pre-train the ARL strategy in offline environment firstly and deploy it in real data center environment for online testing. The test results show that the ARL has significant advantages in maintaining temperature stability while reducing energy consumption, and the PPO algorithm performs better than the DDPG algorithm. Compared with the PID algorithm, the PPO algorithm can save energy by 5.27%, and the temperature control effect can be improved by 3.27%,which indicates the feasibility of the ARL for implementation in real data centers.

Adaptive Optimization of Hyper-Parameters for Robotic Manipulation through Evolutionary Reinforcement Learning

Deep Reinforcement Learning applications are growing due to their capability of teaching the agent any task autonomously and generalizing the learning. However, this comes at the cost of a large number of samples and interactions with the environment. Moreover, the robustness of learned policies is usually achieved by a tedious tuning of hyper-parameters and reward functions. In order to address this issue, this paper proposes an evolutionary RL algorithm for the adaptive optimization of hyper-parameters. The policy is trained using an on-policy algorithm, Proximal Policy Optimization (PPO), coupled with an evolutionary algorithm. The achieved results demonstrate an improvement in the sample efficiency of the RL training on a robotic grasping task. In particular, the learning is improved with respect to the baseline case of a non-evolutionary agent. The evolutionary agent needs 60\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$60$$\\end{document}% fewer samples to completely learn the grasping task, enabled by the adaptive transfer of knowledge between the agents through the evolutionary algorithm. The proposed approach also demonstrates the possibility of updating reward parameters during training, potentially providing a general approach to creating reward functions.

A Time-Aggregated Model-Free RL Algorithm for Optimal Containment Control of MASs

A Time-Aggregated Model-Free RL Algorithm for Optimal Containment Control of MASs

Reinforcement learning for Multi-Flight Dynamic Pricing

Dynamic pricing is essential for airline revenue management, requiring quick adaptation to fluctuating market environments and complex customer behaviors. This study addresses the Multi-Flight Dynamic Pricing (MFDP) problem, which presents unique challenges due to interdependent demand between multiple flights and high dimensionality. Traditional studies often assume that the demand function modeling customer behavior is either known in advance or follows a predefined structure, failing to capture the dynamic nature of pricing decisions. To fill this gap, we develop deep reinforcement learning (DRL) algorithms—Deep Q-Network (DQN), Advantage Actor-Critic (A2C), Proximal Policy Optimization (PPO), and Trust Region Policy Optimization (TRPO). By formulating the MFDP problem as a Markov Decision Process (MDP), we design an innovative utility function for the Multinomial Logit (MNL) model that captures realistic features of the airline market, such as competition from high-speed rail, the effect of reference fares, and travel time. We compare the performance of our DRL algorithms with traditional algorithms, including Dynamic Programming (DP), Price Pooling (PP), Inventory Pooling (IP), and Inventory and Price Pooling (IPP). Our experiments demonstrate that DRL algorithms alleviate the curse of dimensionality faced by traditional algorithms, expedite the learning process, and deliver satisfactory performance without relying on predefined demand functions. Among these algorithms, TRPO shows superior performance, achieving 99% of the theoretical optimal revenue, proving its adaptability and stability in dynamic pricing applications. We also highlight the importance of considering the null price in the action space of MFDP problems. The larger the market scale, the more pronounced the effect of the null price in accelerating RL algorithm convergence, leading to more efficient computational resource utilization.

Event-Triggered Guarantee Cost Control for Partially Unknown Stochastic Systems via Explorized Integral Reinforcement Learning Strategy.

In this article, an integral reinforcement learning (IRL)-based event-triggered guarantee cost control (GCC) approach is proposed for stochastic systems which are modulated by randomly time-varying parameters. First, with the aid of the RL algorithm, the optimal GCC (OGCC) problem is converted into an optimal zero-sum game by solving a modified Hamilton-Jacobin-Isaac (HJI) equation of the auxiliary system. Moreover, in order to address the stochastic zero-sum game, we propose an on-policy IRL-based control approach involved by the multivariate probabilistic collocation method (MPCM), which can accurately predict the mean value of uncertain functions with randomly time-varying parameters. Furthermore, a novel GCC method, which combines the explorized IRL algorithm and MPCM, is designed to relax the restriction of knowing the system dynamics for the class of stochastic systems. On this foundation, for the purpose of reducing computation cost and avoiding the waste of resources, we propose an event-triggered GCC approach involved with explorized IRL and MPCM by utilizing critic-actor-disturbance neural networks (NNs). Meanwhile, the weight vectors of three NNs are updated simultaneously and aperiodically according to the designed triggering condition. The ultimate boundedness (UB) properties of the controlled systems have been proved by means of the Lyapunov theorem. Finally, the effectiveness of the developed GCC algorithms is illustrated via two simulation examples.

A Multi-Agent RL Algorithm for Dynamic Task Offloading in D2D-MEC Network with Energy Harvesting.

Delay-sensitive task offloading in a device-to-device assisted mobile edge computing (D2D-MEC) system with energy harvesting devices is a critical challenge due to the dynamic load level at edge nodes and the variability in harvested energy. In this paper, we propose a joint dynamic task offloading and CPU frequency control scheme for delay-sensitive tasks in a D2D-MEC system, taking into account the intricacies of multi-slot tasks, characterized by diverse processing speeds and data transmission rates. Our methodology involves meticulous modeling of task arrival and service processes using queuing systems, coupled with the strategic utilization of D2D communication to alleviate edge server load and prevent network congestion effectively. Central to our solution is the formulation of average task delay optimization as a challenging nonlinear integer programming problem, requiring intelligent decision making regarding task offloading for each generated task at active mobile devices and CPU frequency adjustments at discrete time slots. To navigate the intricate landscape of the extensive discrete action space, we design an efficient multi-agent DRL learning algorithm named MAOC, which is based on MAPPO, to minimize the average task delay by dynamically determining task-offloading decisions and CPU frequencies. MAOC operates within a centralized training with decentralized execution (CTDE) framework, empowering individual mobile devices to make decisions autonomously based on their unique system states. Experimental results demonstrate its swift convergence and operational efficiency, and it outperforms other baseline algorithms.

Integral reinforcement learning-based angular acceleration autopilot for high dynamic flight vehicles

During the synthesis of acceleration autopilots for high dynamic flight vehicles (HDFV), autopilots with feedback of angular acceleration (AFAA) have become more perspective with stringent requirements on response speed and high maneuverability, compared with autopilots with feedback of angular rate (AFAR). Integral reinforcement learning (IRL) method has now proved to be an effective technique for adaptive optimal control of partially unknown nonlinear systems. In this paper, a novel data-driven IRL algorithm with “actor–critic” structure is proposed for HDFV utilizing AFAA. As an advanced model-free approach, “actor–critic” based IRL algorithm learns optimal behaviors by observing the real-time responses from the environment under the action of nonoptimal control policies. Instead of solving algebraic Riccati equation directly, the control policy updates online via the solution of proposed IRL Bellman equation with sensed quantities. Numerical simulation is carried out to validate the effectiveness of proposed online IRL-based angular acceleration autopilot for HDFVs. Besides, the tracking performance under different wave commands, the robustness against parameter uncertainties and the noise attenuation capacity between classical optimal tracking approach and proposed IRL method are analyzed for AFAR and AFAA, respectively. Simulation results show that, angular acceleration autopilot with proposed integral RL algorithm possesses better tracking performance against various disturbances.

Transfer Reinforcement Learning for Combinatorial Optimization Problems

Reinforcement learning is an important technique in various fields, particularly in automated machine learning for reinforcement learning (AutoRL). The integration of transfer learning (TL) with AutoRL in combinatorial optimization is an area that requires further research. This paper employs both AutoRL and TL to effectively tackle combinatorial optimization challenges, specifically the asymmetric traveling salesman problem (ATSP) and the sequential ordering problem (SOP). A statistical analysis was conducted to assess the impact of TL on the aforementioned problems. Furthermore, the Auto_TL_RL algorithm was introduced as a novel contribution, combining the AutoRL and TL methodologies. Empirical findings strongly support the effectiveness of this integration, resulting in solutions that were significantly more efficient than conventional techniques, with an 85.7% improvement in the preliminary analysis results. Additionally, the computational time was reduced in 13 instances (i.e., in 92.8% of the simulated problems). The TL-integrated model outperformed the optimal benchmarks, demonstrating its superior convergence. The Auto_TL_RL algorithm design allows for smooth transitions between the ATSP and SOP domains. In a comprehensive evaluation, Auto_TL_RL significantly outperformed traditional methodologies in 78% of the instances analyzed.

An Empirical Study of Different Reinforcement Learning Algorithms for Resource Allocation in Cloud Computing

Regarding the increasing importance of cloud computing in modern IT architecture, it is crucial to create highly efficient resource allocation algorithms. This work conducts an empirical investigation into the utilisation of several reinforcement learning methods for optimising resource allocation in cloud computing environments. Our objective is to assess the efficiency of RL algorithms in a dynamic environment with fluctuating workloads, focusing on resource utilisation, cost effectiveness, and optimality. This research examines the effects of altering cloud settings in order to integrate theoretical reinforcement learning concepts into a practical resource management system. In the literature review, we consider the traditional resource allocation techniques and their inability to accommodate the changing demand. We additionally analyse the available research employing machine learning approaches, paying special attention to RL in cloud computing resource distribution. The methodology describes the research design, detailing the employed RL algorithms Q-learning, Deep Q Networks (DQN), and Proximal Policy Optimization (PPO). We explain the data collection procedure that involves different workloads and also situations to mimic real environments. Performance of each RL algorithm is presented in the experimental results based on the resource utilization, cost efficiency and also system responsiveness. Q-learning, DQN and PPO are being tested which provides a better understanding of their pros and cons. Discussion that follows the Interprets results of these findings bringing to light many challenges along the way as well as possible directions for future inquiry. Therefore, this research fills in the evolving landscape of cloud computing by demonstrating RL algorithms’ adaptability and effectiveness regarding resource allocation challenges under the dynamic environments.

APPA-3D: an autonomous 3D path planning algorithm for UAVs in unknown complex environments

Due to their high flexibility, low cost, and ease of handling, Unmanned Aerial Vehicles (UAVs) are often used to perform difficult tasks in complex environments. Stable and reliable path planning capability is the fundamental demand for UAVs to accomplish their flight tasks. Most researches on UAV path planning are carried out under the premise of known environmental information, and it is difficult to safely reach the target position in the face of unknown environment. Thus, an autonomous collision-free path planning algorithm for UAVs in unknown complex environments (APPA-3D) is proposed. An anti-collision control strategy is designed using the UAV collision safety envelope, which relies on the UAV's environmental awareness capability to continuously interact with external environmental information. A dynamic reward function of reinforcement learning combined with the actual flight environment is designed and an optimized reinforcement learning action exploration strategy based on the action selection probability is proposed. Then, an improved RL algorithm is used to simulate the UAV flight process in unknown environment, and the algorithm is trained by interacting with the environment, which finally realizes autonomous collision-free path planning for UAVs. The comparative experimental results in the same environment show that APPA-3D can effectively guide the UAV to plan a safe and collision-free path from the starting point to the target point in an unknown complex 3D environment.

On principle of optimality for safety-constrained Markov Decision Process and p-Safe Reinforcement Learning ⋆

We study optimality for the safety-constrained Markov decision process which is the underlying framework for safe reinforcement learning. Specifically, we consider an undiscounted safety-constrained Markov decision process subject to random stopping times. The decision maker's goal is to reach a goal state while avoiding unsafe states with certain probabilistic guarantees. Therefore the underlying Markov chain for any control policy will be Multichain or non-ergodic since by definition there exists a goal set and an unsafe set. Bellman's principle of optimality does not hold for such a safety-constrained Markov decision process in a Multichain setting as highlighted by a counterexample. We resolve the aforementioned counterexample by considering a zero-sum game setting between the policy and the Lagrange multiplier vector. Under suitable assumptions regarding the existence of admissible policy, we propose an off-policy RL algorithm for learning an optimal policy that satisfies the probabilistic safety guarantees. After that, we present the finite time error bound of the proposed RL algorithm. Lastly, we present simulation results of the aforementioned RL algorithm on a robot in a grid world setting.

Optimization of a Redox-Flow Battery Simulation Model Based on a Deep Reinforcement Learning Approach

Vanadium redox-flow batteries (VRFBs) have played a significant role in hybrid energy storage systems (HESSs) over the last few decades owing to their unique characteristics and advantages. Hence, the accurate estimation of the VRFB model holds significant importance in large-scale storage applications, as they are indispensable for incorporating the distinctive features of energy storage systems and control algorithms within embedded energy architectures. In this work, we propose a novel approach that combines model-based and data-driven techniques to predict battery state variables, i.e., the state of charge (SoC), voltage, and current. Our proposal leverages enhanced deep reinforcement learning techniques, specifically deep q-learning (DQN), by combining q-learning with neural networks to optimize the VRFB-specific parameters, ensuring a robust fit between the real and simulated data. Our proposed method outperforms the existing approach in voltage prediction. Subsequently, we enhance the proposed approach by incorporating a second deep RL algorithm—dueling DQN—which is an improvement of DQN, resulting in a 10% improvement in the results, especially in terms of voltage prediction. The proposed approach results in an accurate VFRB model that can be generalized to several types of redox-flow batteries.

A new optimal adaptive backstepping control approach for nonlinear systems under deception attacks via reinforcement learning

In this paper, a new optimal adaptive backstepping control approach for nonlinear systems under deception attacks via reinforcement learning is presented in this paper. The existence of nonlinear terms in the studied system makes it very difficult to design the optimal controller using traditional methods. To achieve optimal control, RL algorithm based on critic–actor architecture is considered for the nonlinear system. Due to the significant security risks of network transmission, the system is vulnerable to deception attacks, which can make all the system state unavailable. By using the attacked states to design coordinate transformation, the harm brought by unknown deception attacks has been overcome. The presented control strategy can ensure that all signals in the closed-loop system are semi-globally ultimately bounded. Finally, the simulation experiment is shown to prove the effectiveness of the strategy.

GRI: General Reinforced Imitation and Its Application to Vision-Based Autonomous Driving

Deep reinforcement learning (DRL) has been demonstrated to be effective for several complex decision-making applications, such as autonomous driving and robotics. However, DRL is notoriously limited by its high sample complexity and its lack of stability. Prior knowledge, e.g., as expert demonstrations, is often available but challenging to leverage to mitigate these issues. In this paper, we propose General Reinforced Imitation (GRI), a novel method which combines benefits from exploration and expert data and is straightforward to implement over any off-policy RL algorithm. We make one simplifying hypothesis: expert demonstrations can be seen as perfect data whose underlying policy gets a constant high reward. Based on this assumption, GRI introduces the notion of offline demonstration agent. This agent sends expert data which are processed both concurrently and indistinguishably with the experiences coming from the online RL exploration agent. We show that our approach enables major improvements on camera-based autonomous driving in urban environments. We further validate the GRI method on Mujoco continuous control tasks with different off-policy RL algorithms. Our method ranked first on the CARLA Leaderboard and outperforms World on Rails, the previous state-of-the-art method, by 17%.

Interpersonal trust modelling through multi-agent Reinforcement Learning

Many existing approaches to model and compute trust in a quantitative way rely on ranking, rating or assessments of agents by other agents. Even though reputation is related with trust, it does not capture all its characteristics. In parallel, many works in neuroscience shows evidence about interpersonal trust being an associative learning process encoded in the human brain. Inspired by other subjects such as Cognitive Processing/Dopamine, where Reinforcement Learning algorithms have served to model those phenomena, we propose a model for trust dynamics based on a multi-agent RL algorithm. We corroborate some trust concepts developed in social sciences within a quantitative framework. We do also propose and assess some metrics for a better understanding about the relation between the trust behaviour and the performance of the agents. Finally, we show that Trust, as described by our proposal, can serve to accelerate learning.

Goal-Conditioned Reinforcement Learning With Disentanglement-Based Reachability Planning

Goal-Conditioned Reinforcement Learning (GCRL) can enable agents to spontaneously set diverse goals to learn a set of skills. Despite the excellent works proposed in various fields, reaching distant goals in temporally extended tasks remains a challenge for GCRL. Current works tackled this problem by leveraging planning algorithms to plan intermediate subgoals to augment GCRL. Their methods need two crucial requirements: (i) a state representation space to search valid subgoals, and (ii) a distance function to measure the reachability of subgoals. However, they struggle to scale to high-dimensional state space due to their non-compact representations. Moreover, they cannot collect high-quality training data through standard GC policies, which results in an inaccurate distance function. Both affect the efficiency and performance of planning and policy learning. In the paper, we propose a goal-conditioned RL algorithm combined with Disentanglement-based Reachability Planning (REPlan) to solve temporally extended tasks. In REPlan, a Disentangled Representation Module (DRM) is proposed to learn compact representations which disentangle robot poses and object positions from high-dimensional observations in a self-supervised manner. A simple REachability discrimination Module (REM) is also designed to determine the temporal distance of subgoals. Moreover, REM computes intrinsic bonuses to encourage the collection of novel states for training. We evaluate our REPlan in three vision-based simulation tasks and one real-world task. The experiments demonstrate that our REPlan significantly outperforms the prior state-of-the-art methods in solving temporally extended tasks.