DRL-based Framework Research Articles

Review Paper on Deep RL for Video Summarization

In recent years, the explosive growth of video content has heightened the need for efficient summarization techniques to distill lengthy videos into concise, informative summaries. Traditional approaches to video summarization often rely on heuristic-based methods or supervised learning techniques, which can be limited by their reliance on predefined features or extensive labeled datasets. To address these limitations, this paper explores the application of Deep Reinforcement Learning (DRL) for video summarization. DRL offers a dynamic framework where an agent learns to optimize summarization strategies through interaction with the video content, enabling adaptive and context-aware summarization. We propose a novel DRL-based framework that combines convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to extract and represent temporal features from video frames. Actor- critic architecture is utilized, where the actor generates candidate summaries and the critic evaluates their quality based on a reward function designed to balance in formativeness and brevity. We introduce a new reward function that incorporates both content relevance and diversity to encourage the generation of summaries that effectively capture key moments and maintain narrative coherence. Experimental results on benchmark video datasets demonstrate that our DRL-based approach significantly outperforms traditional methods in terms of both summary quality and user satisfaction. The proposed method not only achieves state-of-the-art performance but also offers greater flexibility and adaptability to diverse video content. This work highlights the potential of DRL in advancing video summarization and opens avenues for future research in optimizing video content extraction and representation

Dynamic RIS partitioning in NOMA systems using deep reinforcement learning

The rapid evolution of wireless communication technologies necessitates innovative solutions to meet the increasing performance requirements of future networks, particularly in terms of spectral efficiency, energy efficiency, and computational efficiency. Reconfigurable Intelligent Surfaces (RIS) and Non-Orthogonal Multiple Access (NOMA) are emerging as promising technologies to enhance wireless communication systems. This paper explores the dynamic partitioning of RIS elements in NOMA systems using Deep Reinforcement Learning (DRL) to optimize resource allocation and overall system performance. We propose a novel DRL-based framework that dynamically adjusts the partitioning of RIS elements to maximize the achievable sum rate and ensure fair resource distribution among users. Our architecture leverages the flexibility of RIS to create an intelligent radio environment, while NOMA enhances spectral efficiency. The DRL model is trained online, adapting to real-time changes in the communication environment. Empirical results demonstrate that our approach closely approximates the performance of the optimal iterative algorithm (exhaustive search) while reducing computational time by up to 90 percent. Furthermore, our method eliminates the need for an offline training phase, providing a significant advantage in dynamic environments by removing the requirement for retraining with every environmental change. These findings highlight the potential of DRL-based dynamic partitioning as a viable solution for optimizing RIS-aided NOMA systems in future wireless networks.

BD-TTS: A blockchain and DRL-based framework for trusted task scheduling in edge computing

Edge computing migrates tasks to the edge of the network for execution, which can provide users with lower latency and better quality of service (QoS). However, due to the complexity and dynamism of edge environments, many task scheduling algorithms in edge computing face challenges in achieving real-time scheduling, while there are also trust issues in heterogeneous and dynamic edge environments. To tackle these issues, we introduce a novel framework for trusted task scheduling in edge computing, leveraging blockchain and deep reinforcement learning (DRL) technologies, named BD-TTS. Specifically, we design a blockchain-based trust management scheme tailored for task scheduling in edge computing. The scheme uses blockchain to store, propagate, and update trust information in a decentralized manner to evaluate the trustworthiness of edge servers. In addition, to assign tasks to edge servers with higher trust values for execution, we introduce a DRL-driven task scheduling algorithm. The algorithm dynamically schedules tasks in real-time based on fluctuations in the trust values of edge servers. The experimental results show that compared to other baseline approaches, BD-TTS effectively reduces the number of tasks assigned to malicious edge servers by over 64.4%, reduces the average task response time by at least 13.9%, and improves the success rate by more than 14.7%.

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Deep reinforcement learning approach to estimate the energy-mix proportion for secure operation of converter-dominated power system

The dynamic evolution of the modern energy landscape has forced the integration of renewable energy sources (RESs) into power generation, catalyzing a paradigm shift towards converter-dominated power systems. This paper addresses the imperative challenge of optimizing power-generating technologies within these complex systems to ensure both stability and sustainability. The underlying issue lies in effectively managing the energy-mix proportion—balancing the total electricity generated with the overall system load—while accounting for the intermittent nature of RESs. Motivated by the insistent need for adaptive and efficient power system management, this paper presents a pioneering deep reinforcement learning (DRL) approach within modern power system applications. Leveraging the Deep Q-Network (DQN) paradigm, the proposed methodology estimates the energy-mix proportion through a data-driven lens. By integrating MATLAB/Simulink and Python libraries, offline training and online testing validate the approach's applicability in real-world scenarios. Results from comprehensive experiments on an IEEE-9 bus system underscore the efficacy of the DRL-based framework. Notably, the online short circuit level (SCL) emerges as a robust indicator of power system security, a significant innovation in stability assessment. The model demonstrates remarkable responsiveness to load fluctuations, optimizing energy generation and respecting operational constraints. Furthermore, the adaptability of grid-forming (GFM) and grid-following (GFL) converters is showcased, highlighting their resilience in converter-dominated power systems. The study offers a promising avenue for future research and underscores the potential of DRL in power system optimization and operation for a sustainable energy future.

CFWS: DRL-Based Framework for Energy Cost and Carbon Footprint Optimization in Cloud Data Centers

CFWS: DRL-Based Framework for Energy Cost and Carbon Footprint Optimization in Cloud Data Centers

Airfoils optimization based on deep reinforcement learning to improve the aerodynamic performance of rotors

Airfoil optimization is the key to improving the aerodynamic performance of a rotor. However, conventional optimization approaches cannot modify the airfoil shape intelligently in the way that an aircraft designer would. Because the optimization process is largely uninterpretable and ungeneralizable, the optimization knowledge and experience cannot be extracted and applied to similar optimization tasks. To address these issues, we propose an optimization framework for rotor airfoils based on deep reinforcement learning (DRL). Our DRL-based framework is capable of learning an interpretable and generalizable optimization strategy for alleviating the dynamic stall of a rotor airfoil. First, to enhance the efficiency of airfoil dynamic stall optimization, a deep neural network is trained to predict the dynamic stall hysteresis loops of rotor airfoils as a surrogate model. The asynchronous advantage actor–critic reinforcement learning algorithm is employed to train and learn the optimization strategy for alleviating the dynamic stall of the rotor airfoil. Next, the OA212 rotor airfoil is optimized using the well-trained optimization strategy. The results show that the dynamic stall characteristics of the airfoil are improved after optimization. The lift coefficients of the optimized airfoil are significantly enhanced, and the drag and moment coefficients peaks are reduced by 46.3% and 73.8%, respectively, compared with the baseline airfoil. Then, 20 airfoils are optimized using the well-trained optimization strategy to evaluate the generalizability of the dynamic stall optimization strategy. The results demonstrate that the optimization strategy learned by DRL for the rotor airfoil optimization is generalizable. Finally, the numerical simulation results comparing a rotor with baseline airfoils to one with optimized airfoils demonstrate that the aerodynamic performance of the optimized rotor is superior to that of the baseline rotor.

Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning

The operation of microgrids, i.e., energy systems composed of distributed energy generation, local loads and energy storage capacity, is challenged by the variability of intermittent energy sources and demands, the stochastic occurrence of unexpected outages of the conventional grid and the degradation of the Energy Storage System (ESS), which is strongly influenced by its operating conditions. To effectively address these challenges, a novel method for combined operation and maintenance management of ESS has been developed. Unlike the currently available solutions, which typically address the one-day-ahead scheduling problem, the present work considers, for the first time, the realistic case of a microgrid in which the ESS degrades and unexpected outages of the conventional grid can occur along the long-time horizons of the entire microgrid lifetimes. The proposed method, which is based on deep reinforcement learning, is tested on a simulated grid-connected microgrid of a residential building equipped with photovoltaic modules and an ESS. The method outperforms other state-of-the-art approaches based on heuristics and metaheuristics by increasing the profit by 15% and reducing the average number of ESS replacements during its lifetime. Therefore, it can be concluded that the proposed DRL-based framework allows achieving prescriptive maintenance since the suggested actions are optimal from the point of view of effectively maximizing the profit and minimizing the maintenance interventions over the entire lifetime of the microgrid.

On Attack-Resilient Service Placement and Availability in Edge-Enabled IoV Networks

Achieving network resilience in terms of attack tolerance and service availability is critically important for Internet of Vehicles (IoV) networks where vehicles require assistance in sensitive and safety-critical applications like driving. It is particularly challenging in time-varying conditions of IoV traffic. In this paper, we study an attack-resilient optimal service placement problem to ensure disruption-free service availability to the users in edge-enabled IoV network. Our work aims to improve the user experience while minimizing the delay and simultaneously considering efficient utilization of limited edge resources. First, an optimal service placement is performed while considering traffic dynamicity and meeting the service requirements with the use of a deep reinforcement learning (DRL) framework. Next, an optimal secondary mapping and service recovery placements are performed to account for the attacks/failures at the edge. The use of DRL framework helps to adapt to dynamically varying IoV traffic and service demands. In this work, we develop three ILP models and use them in the DRL-based framework to provide attack-resilient service placement and ensure service availability with efficient network performance. Extensive numerical experiments are performed to demonstrate the effectiveness of the proposed approach.

A Novel Deep Reinforcement Learning Based Framework for Gait Adjustment

Nowadays, millions of patients suffer from physical disabilities, including lower-limb disabilities. Researchers have adopted a variety of physical therapies based on the lower-limb exoskeleton, in which it is difficult to adjust equipment parameters in a timely fashion. Therefore, intelligent control methods, for example, deep reinforcement learning (DRL), have been used to control the medical equipment used in human gait adjustment. In this study, based on the key-value attention mechanism, we reconstructed the agent’s observations by capturing the self-dependent feature information for decision-making in regard to each state sampled from the replay buffer. Moreover, based on Softmax Deep Double Deterministic policy gradients (SD3), a novel DRL-based framework, key-value attention-based SD3 (AT_SD3), has been proposed for gait adjustment. We demonstrated the effectiveness of our proposed framework in gait adjustment by comparing different gait trajectories, including the desired trajectory and the adjusted trajectory. The results showed that the simulated trajectories were closer to the desired trajectory, both in their shapes and values. Furthermore, by comparing the results of our experiments with those of other state-of-the-art methods, the results proved that our proposed framework exhibited better performance.

DRL-Based Partial Offloading for Maximizing Sum Computation Rate of Wireless Powered Mobile Edge Computing Network

The advanced Internet of Things (IoT) enables more and more interactions between people and machines in the emerging applications, which rely on real-time communication and computing. However, the limited battery capacity and low computing capacity of IoT nodes can hardly support high-performance computing applications. The integration of wireless power transmission (WPT) and mobile edge computing (MEC) is a feasible and promising solution to address the energy shortage and computing capacity limitation of IoT nodes by harvesting radio frequency signal’s energy and offloading the nodes’ computation tasks to edge computing servers (ECSs). In this work, we focus on the wireless powered MEC network with an ECS and multiple edge devices (EDs), and study the joint optimization of WPT duration, transmission time allocation of each ED and partial offloading decision to maximize the sum computation rate. First, we formulate this as a non-convex problem which is hard to solve. Second, to conquer this problem, we decompose the original offloading problem into the sub-problem of optimizing the offloading time allocation among EDs and the proportion of harvested energy allocated for offloading at each ED under a given WPT duration and the top-problem of optimizing the WPT duration. Finally, we design an online DRL-based framework where one DNN together with its exploration strategy and training strategy is adopted to learn the near-optimal WPT duration and an efficient optimal algorithm is designed to solve the sub-problem. Numerical results show that the DRL-based offloading algorithm achieves the near-maximal sum computation rate while greatly reducing the processing time by at least three orders of magnitude compared with using the solver CVX for the sub-problem and the DNN for the top-problem.

Deep Reinforcement Learning-Based End-to-End Control for UAV Dynamic Target Tracking.

Uncertainty of target motion, limited perception ability of onboard cameras, and constrained control have brought new challenges to unmanned aerial vehicle (UAV) dynamic target tracking control. In virtue of the powerful fitting ability and learning ability of the neural network, this paper proposes a new deep reinforcement learning (DRL)-based end-to-end control method for UAV dynamic target tracking. Firstly, a DRL-based framework using onboard camera image is established, which simplifies the traditional modularization paradigm. Secondly, neural network architecture, reward functions, and soft actor-critic (SAC)-based speed command perception algorithm are designed to train the policy network. The output of the policy network is denormalized and directly used as speed control command, which realizes the UAV dynamic target tracking. Finally, the feasibility of the proposed end-to-end control method is demonstrated by numerical simulation. The results show that the proposed DRL-based framework is feasible to simplify the traditional modularization paradigm. The UAV can track the dynamic target with rapidly changing of speed and direction.

Cognitive Conformal Antenna Array Exploiting Deep Reinforcement Learning Method

A cognitive antenna array, which is designed by using deep reinforcement learning (DRL) is proposed in this article to adapt to the complex electromagnetic environment. Specifically, the phased array antenna is utilized as the manipulatable component to achieve the characteristic of beam steering with the help of the DRL algorithm. We begin by establishing a DRL-based framework, which is comprised of a microprogrammed control unit, power divider, digital phase shifter, and the patch antenna array. In the DRL algorithm, the desired beam steering is obtained through trial-and-error interactions with the environment, which is required to observe predefined rewards based on the current state and action. Next, the system is trained to perform beam steering, and a set of hyperparameters of the deep neural network are obtained and stored for practical usage. A good agreement is achieved between the simulated and measured radiation patterns of the planar phased array antenna, which validates the DRL-based phase distribution regulation algorithm. Finally, the algorithm is implemented in the design process of a conformal phased array antenna, and it is shown that the measured radiation performance of the array is satisfactory for different beam scan angles.

Hierarchical Multi-Agent DRL-Based Framework for Joint Multi-RAT Assignment and Dynamic Resource Allocation in Next-Generation HetNets

This article considers the problem of cost-aware downlink sum-rate maximization via joint optimal radio access technologies (RATs) assignment and power allocation in next-generation heterogeneous wireless networks (HetNets). We consider a future HetNet comprised of multi-RATs and serving multi-connectivity edge devices (EDs), and we formulate the problem as a mixed-integer non-linear programming (MINP) problem. Due to the high complexity and combinatorial nature of this problem and the difficulty to solve it using conventional methods, we propose a hierarchical multi-agent deep reinforcement learning (DRL)-based framework, called DeepRAT, to solve it efficiently and learn system dynamics. In particular, the DeepRAT framework decomposes the problem into two main stages; the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RATs-EDs assignment stage</i> , which implements a single-agent Deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> Network (DQN) algorithm, and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">power allocation stage</i> , which utilizes a multi-agent Deep Deterministic Policy Gradient (DDPG) algorithm. Using simulations, we demonstrate how the various DRL agents efficiently interact to learn system dynamics and derive the global optimal policy. Furthermore, our simulation results show that the proposed DeepRAT algorithm outperforms existing state-of-the-art heuristic approaches in terms of network utility. Finally, we quantitatively show the ability of the DeepRAT model to quickly and dynamically adapt to abrupt changes in network dynamics, such as EDs’ mobility.

Deep-Reinforcement-Learning-Based Predictive Maintenance Model for Effective Resource Management in Industrial IoT

Unplanned breakdown of critical equipment interrupts production throughput in Industrial IoT (IIoT), and data-driven Predictive Maintenance (PdM) becomes increasingly important for companies seeking a competitive business advantage. Manufacturers, however, are constantly faced with the onerous challenge of manually allocating suitably competent manpower resources in the event of an unexpected machine breakdown. Furthermore, human error has a negative rippling impact on both overall equipment downtime and production schedules. In this paper, we formulate the complex resource management problem as a resource optimisation problem to determine if a model-free Deep Reinforcement Learning (DRL) based PdM framework can be used to automatically learn an optimal decision-policy from a stochastic environment. Unlike the existing PdM frameworks, our approach considers PdM sensor information and the resources of both physical equipment and human as part of the optimisation problem. The proposed DRL-based framework and Proximal Policy Optimisation Long Short Term Memory (PPO-LSTM) model are evaluated alongside baselines results from human participants using a maintenance repair simulator. Empirical results indicate that our PPO-LSTM efficiently learns the optimal decision-policy for the resource management problem, outperforming comparable DRL methods and human participants by 53% and 65% respectively. Overall, the simulation results corroborate the proposed DRL-based PdM framework’s superiority in terms of convergence efficiency, simulation performance and flexibility.

Deep Reinforcement Learning Aided Cell Outage Compensation Framework in 5G Cloud Radio Access Networks

As one of the key technologies of 5G, Cloud Radio Access Networks (C-RAN) with cloud BBUs (Base Band Units) pool architecture and distributed RRHs (Remote Radio Heads) can provide the ubiquitous services. When failure occurs at RRH, it can’t be alleviated in time and will lead to a significant drop in network performance. Therefore, the cell outage compensation (COC) problem for RRH in 5G C-RAN is very important. Although deep reinforcement learning (DRL) has been applied to many scenarios related to the self-organizing network (SON), there are fewer applications for cell outage compensation. And most intelligent algorithms are hard to obtain globally optimized solutions. In this paper, aiming at the cell outage scenario in C-RAN with the goal of maximizing the energy efficiency, connectivity of RRH while meeting service quality demands of each compensation user, a framework based on DRL is presented to solve it. Firstly, compensation users are allocated to adjacent RRHs by using the K-means clustering algorithm. Secondly, DQN is used to find the antenna downtilt and the power allocated to compensation users. Comparing to different genetic algorithms, simulation result shows that the proposed framework converges quickly and tends to be stable, and reaches 95% of the maximum target value. It verifies the efficiency of the DRL-based framework and its effectiveness in meeting user requirements and handling cell outage compensation.

A Hybrid Learning Framework for Service Function Chaining Across Geo-Distributed Data Centers

Service function chaining (SFC) focuses mainly on deploying various network functions in geographically distributed data centers and providing interconnect routing among them. Traditional (convex optimization-based) SFC algorithms exhibit some disadvantages on the scalability and accuracy. Recently, researches have shown the effectiveness of deep reinforcement learning (DRL) in the field of SFC. However, current DRL-based algorithms possess an extremely large action space, which leads to slow convergence and poor scalability. Some researchers relieve this issue by reformulating the SFC problem, which usually results in low utilization and high cost. To address this issue, we develop a hybrid DRL-based framework which decouples the VNF deployment and flow routing into different modules. In the proposed framework, a DRL agent is only responsible for learning the policy of VNF deployment. We customize the structure of the agent base on deep deterministic policy gradient (DDPG) and adopt several techniques to improve the learning efficiency, such as adaptive parameter noise, wolpertinger policy, and prioritized experience replay. The flow routing is conducted in a game-based module (GBM). We design a decentralized routing algorithm for the GBM to address the scalability. The end-to-end latency of flows is minimized while the resource capacity and location constraints are satisfied. During the learning process of the proposed framework, the DRL agent improves its deployment policy with the reward from the GBM (the value of reward depends on flow routing). Thus, the VNF deployment and flow routing are still jointly optimized. Compared to existing DRL-learning algorithms, the proposed hybrid DRL framework can achieve a lower cost since 1) the action space is significantly reduced due to flow routing decoupling; 2) the flow routing procedure is more efficient (the GBM adopts model-based information, e.g., the gradient). Through trace-driven simulations, we show the efficiency of our algorithm compared to existing DRL-based algorithms.