Reinforcement Learning-based Framework Research Articles

JamBIT: RL-based framework for disrupting adversarial information in battlefields

During battlefield operations, military radios (hereafter nodes) exchange information among various units using a mobile ad-hoc network (MANET) due to its infrastructure-less and self-healing capabilities. Adversarial cyberwarfare plays a crucial role in modern combat by disrupting communication between critical nodes (i.e., nodes mainly responsible for propagating important information) to gain dominance over the opposing side. However, determining critical nodes within a complex network is an NP-hard problem. This paper formulates a mathematical model to identify important links and their connected nodes, and presents JamBIT, a reinforcement learning-based framework with an encoder–decoder architecture, for efficiently detecting and jamming critical nodes. The encoder transforms network structures into embedding vectors, while the decoder assigns a score to the embedding vector with the highest reward. Our framework is trained and tested on custom-built MANET topologies using the Named Data Networking (NDN) protocol. JamBIT has been evaluated across various scales and weighting methods for both connected node and network dismantling problems. Our proposed method outperformed existing RL-based baselines, with a 24% performance gain for smaller topologies (50-100 nodes) and 8% for larger ones (400-500 nodes) in connected node problems, and a 7% gain for smaller topologies and 15% for larger ones in network dismantling problems.

Reinforcement learning-based framework for whale rendezvous via autonomous sensing robots.

Rendezvous with sperm whales for biological observations is made challenging by their prolonged dive patterns. Here, we propose an algorithmic framework that codevelops multiagent reinforcement learning-based routing (autonomy module) and synthetic aperture radar-based very high frequency (VHF) signal-based bearing estimation (sensing module) for maximizing rendezvous opportunities of autonomous robots with sperm whales. The sensing module is compatible with low-energy VHF tags commonly used for tracking wildlife. The autonomy module leverages in situ noisy bearing measurements of whale vocalizations, VHF tags, and whale dive behaviors to enable time-critical rendezvous of a robot team with multiple whales in simulation. We conducted experiments at sea in the native habitat of sperm whales using an "engineered whale"-a speedboat equipped with a VHF-emitting tag, emulating five distinct whale tracks, with different whale motions. The sensing module shows a median bearing error of 10.55° to the tag. Using bearing measurements to the engineered whale from an acoustic sensor and our sensing module, our autonomy module gives an aggregate rendezvous success rate of 81.31% for a 500-meter rendezvous distance using three robots in postprocessing. A second class of fielded experiments that used acoustic-only bearing measurements to three untagged sperm whales showed an aggregate rendezvous success rate of 68.68% for a 1000-meter rendezvous distance using two robots in postprocessing. We further validated these algorithms with several ablation studies using a sperm whale visual encounter dataset collected by marine biologists.

A learning-based influence maximization framework for complex networks via K-core hierarchies and reinforcement learning

Influence maximization (IM) is a critical aspect of complex network analysis, as it holds considerable commercial value across various domains such as recommendation systems and viral marketing. Its objective is to identify a set of seed nodes that can initiate the most extensive cascade of influence within the network. However, IM encounters significant challenges due to its NP-hard nature of accurately identifying the optimal seed nodes and estimating the spread of influence. Although recent learning-based IM methods have shown promising results, there are still challenges related to scalability, efficiency, and avoiding influence overlap that need to be addressed. This paper proposes CoreQ, a novel reinforcement learning-based framework that leverages K-core hierarchies to address IM challenges and make advised decisions to learn the optimal policy to select seed nodes. CoreQ significantly enhances IM by reducing computational costs and effectively addressing the influence overlap problem, making it a scalable and efficient solution. CoreQ first utilizes K-core decomposition to analyze the network structure and extract hierarchical features that guide the process of seed node identification. Then, it employs a novel maximum likelihood-based approach to identify a sufficient number of candidate seed nodes from K-core hierarchies. Finally, it leverages Q-learning’s decision-making capabilities to optimize a policy for learning the most effective seed selection strategy. The experimental results demonstrate that CoreQ not only outperforms state-of-the-art IM methods in terms of influence spread under the weighted independent cascade model but also surpasses other competing learning-based IM methods in terms of time efficiency.

A deep reinforcement learning-based scheduling framework for real-time workflows in the cloud environment

Benefiting from the flexibility, scalability and many other advantages of cloud computing, an increasing number of research institutions and enterprises are using cloud platforms to deploy their scientific or commercial workflow applications. From the perspective of cloud service providers, how to schedule workflows reasonably to improve user satisfaction and resource utilization is a big challenge. This challenge becomes more prominent in a dynamic cloud environment, where the workload of user-submitted workflows is constantly changing and unpredictable. Traditional scheduling methods designed for fixed workflows or real-time independent tasks are not applicable in such scenarios. In this paper, we present an online scheduling framework for multiple real-time workflows in a dynamic environment. In this framework, the scheduler employs a deep reinforcement learning based algorithm (R-DQN) to assign each task in workflows to a suitable virtual machine instance according to the current situation. Its goal is to achieve optimal task scheduling and resource allocation, so as to minimize workflow makespan and maximize resource utilization. We provide a comprehensive overview of the design and implementation of our approach, and we conduct experiments using various workflow benchmarks with different structures and scales on the WorkflowSim platform. The experimental results demonstrate that our algorithm significantly outperforms the compared algorithms in term of makespan and other metrics. Moreover, our algorithm demonstrates strong performance in generality, robustness and scalability.

Reinforcement Learning-Based Control Sequence Optimization for Advanced Reactors

The last decade has seen the development and application of data-driven methods taking off in nuclear engineering research, aiming to improve the safety and reliability of nuclear power. This work focuses on developing a reinforcement learning-based control sequence optimization framework for advanced nuclear systems, which not only aims to enhance flexible operations, promoting the economics of advanced nuclear technology, but also prioritizing safety during normal operation. At its core, the framework allows the sequence of operational actions to be learned and optimized by an agent to facilitate smooth transitions between the modes of operations (i.e., load-following), while ensuring that all safety significant system parameters remain within their respective limits. To generate dynamic system responses, facilitate control strategy development, and demonstrate the effectiveness of the framework, a simulation environment of a pebble-bed high-temperature gas-cooled reactor was utilized. The soft actor-critic algorithm was adopted to train a reinforcement learning agent, which can generate control sequences to maneuver plant power output in the range between 100% and 50% of the nameplate power through sufficient training. It was shown in the performance validation that the agent successfully generated control actions that maintained electrical output within a tight tolerance of 0.5% from the demand while satisfying all safety constraints. During the mode transition, the agent can maintain the reactor outlet temperature within ±1.5 °C and steam pressure within 0.1 MPa of their setpoints, respectively, by dynamically adjusting control rod positions, control valve openings, and pump speeds. The results demonstrate the effectiveness of the optimization framework and the feasibility of reinforcement learning in designing control strategies for advanced reactor systems.

CrowdBot: An Open-Environment Robot Management System for On-Campus Services

In contemporary campus environments, the provision of timely and efficient services is increasingly challenging due to limitations in accessibility and the complexity and openness of the environment. Existing service robots, while operational, often struggle with adaptability and dynamic task management, leading to inefficiencies. To overcome these limitations, we introduce CrowdBot, a robot management system that enhances service in campus environments. Our system leverages a hierarchical reinforcement learning-based cloud-edge hybrid scheduling framework (REDIS), for efficient online streaming task assignment and dynamic action scheduling. To verify the REDIS framework, we have developed a digital twin simulation platform, which integrates large language models and hot-swapping technology. This facilitates seamless human-robot interaction, efficient task allocation, and cost-effective execution through the reuse of robot equipment. Our comprehensive simulations corroborate the system's remarkable efficacy, demonstrating significant improvements with a 24.46% reduction in task completion times, a 9.37% decrease in travel distances, and up to a 3% savings in power usage. Additionally, the system achieves a 7.95% increase in the number of tasks completed and a 9.49% reduction in response time. Real-world case studies further affirm CrowdBot's capability to adeptly execute tasks and judiciously recycle resources, thereby offering a smart and viable solution for the streamlined management of campus services.

Reinforcement learning-based control for waste biorefining processes under uncertainty

Waste biorefining processes face significant challenges related to the variability of feedstocks. The supply and composition of multiple feedstocks in these processes can be uncertain, making it difficult to achieve economically feasible and sustainable waste valorization for large-scale production. Here, we introduce a reinforcement learning-based framework that aims to control these uncertainties and improve the efficiency of the process. The framework is tested on an anaerobic digestion process and is found to perform better than traditional control strategies. In the short term, it achieves faster target tracking with increased precision and accuracy, while in the long term, it shows adaptive and robust behavior even under additional seasonal supply variability, meeting downstream demand with high probability. This reinforcement learning-based framework offers a promising and scalable solution to address uncertainty issues in real-world biorefining processes. If implemented, this framework could contribute to sustainable waste management practices globally, making waste biorefining processes more economically viable and environmentally friendly.

Hierarchical Deep Reinforcement Learning-Based Propofol Infusion Assistant Framework in Anesthesia.

This article aims to provide a hierarchical reinforcement learning (RL)-based solution to the automated drug infusion field. The learning policy is divided into the tasks of: 1) learning trajectory generative model and 2) planning policy model. The proposed deep infusion assistant policy gradient (DIAPG) model draws inspiration from adversarial autoencoders (AAEs) and learns latent representations of hypnotic depth trajectories. Given the trajectories drawn from the generative model, the planning policy infers a dose of propofol for stable sedation of a patient under total intravenous anesthesia (TIVA) using propofol and remifentanil. Through extensive evaluation, the DIAPG model can effectively stabilize bispectral index (BIS) and effect site concentration given a potentially time-varying target sequence. The proposed DIAPG shows an increased performance of 530% and 15% when a human expert and a standard reinforcement algorithm are used to infuse drugs, respectively.

Snippet Policy Network V2: Knee-Guided Neuroevolution for Multi-Lead ECG Early Classification.

Early time series classification predicts the class label of a given time series before it is completely observed. In time-critical applications, such as arrhythmia monitoring in ICU, early treatment contributes to the patient's fast recovery, and early warning could even save lives. Hence, in these cases, it is worthy of trading, to some extent, classification accuracy in favor of earlier decisions when the time series data are collected over time. In this article, we propose a novel deep reinforcement learning-based framework, snippet policy network V2 (SPN-V2), for long and varied-length multi-lead electrocardiogram (ECG) early classification. The proposed SNP-V2 contains two main components: snippet representation learning (SRL) and early classification timing learning (ECTL). The SRL is proposed to encode inner-snippet spatial correlations and inter-snippet temporal correlations into the hidden representations of the subsegment (snippet) of the input ECG. ECTL aims to learn a decision agent to classify the time series early and accurately. To optimize the proposed framework, we design a novel knee-guided neuroevolution algorithm (KGNA) to solve cardiovascular diseases' early classification problem, automatically optimizing the proposed SPN-V2 regarding the tradeoff between accuracy and earliness. In addition, we conduct a series of experiments on two real-world ECG datasets. The experimental results show the superiority of the proposed algorithm over the state-of-the-art competing methods.

Automated guided vehicle dispatching and routing integration via digital twin with deep reinforcement learning

The manufacturing industry has witnessed a significant shift towards high flexibility and adaptability, driven by personalized demands. However, automated guided vehicle (AGV) dispatching optimization is still challenging when considering AGV routing with the spatial-temporal and kinematics constraints in intelligent production logistics systems, limiting the evolving industry applications. Against this backdrop, this paper presents a digital twin (DT)-enhanced deep reinforcement learning-based optimization framework to integrate AGV dispatching and routing at both horizontal and vertical levels. First, the proposed framework leverages a digital twin model of the shop floor to provide a simulation environment that closely mimics the actual manufacturing process, enabling the AGV dispatching agent to be trained in a realistic setting, thus reducing the risk of finding unrealistic solutions under specific shop-floor settings and preventing time-consuming trial-and-error processes. Then, the AGV dispatching with the routing problem is modeled as a Markov Decision Process to optimize tardiness and energy consumption. An improved dueling double deep Q network algorithm with count-based exploration is developed to learn a better-dispatching policy by interacting with the high-fidelity DT model that integrates a static path planning agent using A* and a dynamic collision avoidance agent using a deep deterministic policy gradient to prevent the congestion and deadlock. Experimental results show that our method outperforms four state-of-the-art methods with shorter tardiness, lower energy consumption, and better stability. The proposed method provides significant potential to utilize the digital twin and reinforcement learning in the decision-making and optimization of manufacturing processes.

RLPRAF: Reinforcement Learning-Based Proactive Resource Allocation Framework for Resource Provisioning in Cloud Environment

RLPRAF: Reinforcement Learning-Based Proactive Resource Allocation Framework for Resource Provisioning in Cloud Environment

Comprehensive Analysis of Adaptive Soft Actor-Critic Reinforcement Learning-Based Control Framework for Autonomous Driving in Varied Scenarios

Comprehensive Analysis of Adaptive Soft Actor-Critic Reinforcement Learning-Based Control Framework for Autonomous Driving in Varied Scenarios

Deep Reinforcement Learning-Based Detection Framework for False Data Injection Attacks in Power Systems

Deep Reinforcement Learning-Based Detection Framework for False Data Injection Attacks in Power Systems

Deep Progressive Reinforcement Learning-Based Flexible Resource Scheduling Framework for IRS and UAV-Assisted MEC System.

The intelligent reflecting surface (IRS) and unmanned aerial vehicle (UAV)-assisted mobile edge computing (MEC) system is widely used in temporary and emergency scenarios. Our goal is to minimize the energy consumption of the MEC system by jointly optimizing UAV locations, IRS phase shift, task offloading, and resource allocation with a variable number of UAVs. To this end, we propose a flexible resource scheduling (FRES) framework by employing a novel deep progressive reinforcement learning that includes the following innovations. First, a novel multitask agent is presented to deal with the mixed integer nonlinear programming (MINLP) problem. The multitask agent has two output heads designed for different tasks, in which a classified head is employed to make offloading decisions with integer variables while a fitting head is applied to solve resource allocation with continuous variables. Second, a progressive scheduler is introduced to adapt the agent to the varying number of UAVs by progressively adjusting a part of neurons in the agent. This structure can naturally accumulate experiences and be immune to catastrophic forgetting. Finally, a light taboo search (LTS) is introduced to enhance the global search of the FRES. The numerical results demonstrate the superiority of the FRES framework, which can make real-time and optimal resource scheduling even in dynamic MEC systems.

Meta-IRLSOT++: A meta-inverse reinforcement learning method for fast adaptation of trajectory prediction networks

Recent research on pedestrian trajectory prediction based on deep learning has made significant progress. However, the previous methods do not deeply explore the relationship between scene information and trajectory. Moreover, the training model requires massive data and only targets specific scenes, so the prediction performance is poor when the new scene samples are limited. To solve the above problems, a meta-inverse reinforcement learning-based framework, dubbed Meta-IRLSOT++, is proposed in this work. IRLSOT++ improves the solid baseline IRLSOT, and the main contributions are as follows: (1) An inverse reinforcement learning framework is introduced to explore the trajectory–scene association to achieve task-level scene understanding, enhancing the correlation between trajectory and scene. The trajectory heat maps describing pedestrians dynamic characteristics are leveraged to align better the trajectory and scene semantic segmentation predicted by TopFormer. (2) A Transformer-based encoder–decoder network is proposed to fuse the trajectory and plan cues for better generating multi-modal trajectories with multi-head attention. The social graph attention and pseudo-oracle predictor are introduced to capture pedestrians’ social interactions and intent states, ameliorating trajectory prediction performance. (3) Meta-learning is leveraged to achieve collaborative training based on IRLSOT++ to improve the model generalization in new scenes, enabling fast adaptation of trajectory prediction. Experimental results on the Stanford Drone Dataset (SDD) indicate that IRLSOT++ can precisely forecast future trajectories by improving IRLSOT, decreasing Average Displacement Error/ Final Displacement Error (ADE/FDE) values from 9.66/13.05 to 8.36/12.28. Moreover, the meta-learning strategy quickly adapts IRLSOT++ to the new scene, achieving ADE/FDE values of 7.31/11.02 and 15.42/26.55 when Tpred is set to 12 and 24, respectively. Both quantitative and qualitative experimental results show that Meta-IRLSOT++ has accuracy and fast adaptability, which is beneficial for real-world trajectory prediction tasks.

Deep reinforcement learning for proactive spectrum defragmentation in elastic optical networks

The immense growth of Internet traffic calls for advanced techniques to enable the dynamic operation of optical networks, efficient use of spectral resources, and automation. In this paper, we investigate the proactive spectrum defragmentation (SD) problem in elastic optical networks and propose a novel deep reinforcement learning-based framework DeepDefrag to increase spectral usage efficiency. Unlike the conventional, often threshold-based heuristic algorithms that address a subset of the defragmentation-related tasks and have limited automation capabilities, DeepDefrag jointly addresses the three main aspects of the SD process: determining when to perform defragmentation, which connections to reconfigure, and which part of the spectrum to reallocate them to. By considering service attributes, the spectrum occupancy state expressed by several different fragmentation metrics, and the reconfiguration cost, DeepDefrag is able to consistently select appropriate reconfiguration actions over the network lifetime and adapt to changing conditions. Extensive simulation results reveal superior performance of the proposed scheme over a scenario with exhaustive defragmentation and a well-known benchmark heuristic from the literature, achieving lower blocking probability at a smaller defragmentation overhead.

Learning adaptive reaching and pushing skills using contact information.

In this paper, we propose a deep reinforcement learning-based framework that enables adaptive and continuous control of a robot to push unseen objects from random positions to the target position. Our approach takes into account contact information in the design of the reward function, resulting in improved success rates, generalization for unseen objects, and task efficiency compared to policies that do not consider contact information. Through reinforcement learning using only one object in simulation, we obtain a learned policy for manipulating a single object, which demonstrates good generalization when applied to the task of pushing unseen objects. Finally, we validate the effectiveness of our approach in real-world scenarios.

Caching Placement Optimization in UAV-Assisted Cellular Networks: A Deep Reinforcement Learning-Based Framework

Capable of delivering contents offloaded from the base station (BS) to users, unmanned aerial vehicle (UAV) has emerged as a crucial leverage to compensate for terrestrial BSs-based communication. However, the limited storage capacity of the UAV brings challenges to providing low-latency services for users. In this paper, we investigate the caching placement of the UAV for enhancing the timeliness of services. To overcome the unknown content popularity, proximal policy optimization (PPO) is adopted in the proposed algorithm. To be specific, we first propose a modified timeliness model, named effective age of information (EAoI), to comprehensively evaluate the timeliness of services. Then, we employ PPO to build a deep reinforcement learning framework for finding the optimal caching strategy adaptively. Extensive simulation results are provided to verify the superiority of the proposed scheme, in comparison with the conventional schemes.

Multi-objective reinforcement learning-based framework for solving selective maintenance problems in reconfigurable cyber-physical manufacturing systems

ABSTRACT Unlike mass production manufacturing systems, where configurations are rarely changed after the initial design, reconfigurable cyber-physical systems (RCPMS) self-change their structures throughout missions and thus self-adjust production in response to demand requirements. Accordingly, such a paradigm requires enhancing selective maintenance strategy to optimise scheduling maintenance actions, selecting configuration layouts for capacity and product family changes, and achieving maintenance cost reduction and reliability maximisation. This paper is the first to propose a robust model for a selective maintenance problem with imperfect repairs in the RCPMS context. The model also integrates uncertainties originating from the imperfect observations of components' health status. The model's objectives are to maximise the expected reliability and minimise the variance and maintenance cost under maintenance resource constraints. Moreover, we propose a new deep reinforcement learning framework for solving the resulting multi-objective and combinatorial optimisation problem. In addition, we use decision values to enhance the scalarisation process by permitting the priorities of specific objectives to be adjusted after the learning process. Furthermore, we employ Analytical Hierarchy Process to adjust the static priorities with respect to the objective functions and the actual learning context. Finally, broad experiments are conducted to highlight the performance of the proposed model and resolution framework.

Efficient Accelerator/Network Co-Search With Circular Greedy Reinforcement Learning

Recently, accelerator/network co-search has shown great promise in reducing the complexity of co-design and achieving higher model accuracy. Unlike the manual design methodology, it automatically learns the optimal network architecture and corresponding accelerator under the given constraints. However, prior works do not consider the direct feedback of searched accelerators on the network search in each step, which leads to low converge speed and sub-optimal solutions. To address this issue, we propose DAN, a reinforcement learning-based framework for fast and accurate accelerator/network co-search. The fundamental idea is to model the co-search as interleaved network-aware accelerator search (AS) and accelerator-aware network search (NS) using separate RL agents, which improves the performance of AS and NS, and encourages a tight interaction between AS and NS. Experimental results show that our proposed method consistently outperforms single-agent based method in terms of converge speed and performance.