Reinforcement Learning-based Algorithm Research Articles

An intelligent fuzzy reinforcement learning-based routing algorithm with guaranteed latency and bandwidth in SDN: Application of video conferencing services

Online video conferencing services connection by multipoint control units leads to communication bottleneck and high latency. Due to the dynamic traffic in routing connection requests to video conferences, the existing routing algorithms face problems such as communication bottlenecks, high latency, congestion, path redundancy, and local optimum. In recent years, the implementation of video conferencing systems using Software-Defined Networking (SDN) provides new opportunities to improve Quality of Service (QoS) with low latency. In this paper, an Intelligent Fuzzy reinforcement learning-based Routing Algorithm is proposed, which simultaneously Guarantees Latency and Bandwidth for online video conferencing services in SDN (named IFRA-GLB). Fuzzy logic mainly performs online routing between an ingress–egress pair. Meanwhile, through ongoing training, reinforcement learning lowers the average number of hops on the path determined by fuzzy model. Enhancements to the convergence capability of IFRA and reduced reliance on network topology are achieved by adjusting link weights using a weighted shortest path algorithm focused on critical nodes. When the network encounters congestion, a deferral module is applied to assign higher priority to requests with lower resource demands. Experimental results demonstrate that IFRA-GLB significantly improves performance and convergence compared to existing solutions, enhancing the QoS for video conferencing services. Specifically, IFRA-GLB increases the average admission rate by 1.96% for MIRA topology and 2.71% for ANSNET topology with 3.5% lower latency.

Reinforcement learning-based charging cluster determination algorithm for optimal charger placement in wireless rechargeable sensor networks

Wireless power transfer (WPT) provides a promising technology for energy replenishment of wireless rechargeable sensor networks (WRSNs), where wireless chargers can be deployed at fixed locations for charging nodes simultaneously within their effective charging range. Optimal charger placement (OCP) for sustainable operations of WRSN with cheaper charging cost is a challenging and difficult problem due to its NP-completeness in nature. This paper proposes a novel reinforcement learning (RL) based approach for OCP, where the problem is firstly formulated as a charging cluster determination problem with a fixed clustering radius and then tackled by the reinforcement learning-based charging cluster determination (RL-CCD) algorithm. Specifically, nodes are coarsely clustered by the K-Means++ algorithm, with chargers placed at the cluster center. Meanwhile, RL is applied to explore the potential locations of the cluster centers to adjust the center locations and reduce the number of clusters, using the number of nodes in the cluster and the summation of distances between the cluster center and nodes as the reward. Moreover, an experience-strengthening mechanism is introduced to learn the current optimal charging experience. Extensive simulations show that RL-CCD significantly outperforms existing algorithms.

A deep reinforcement learning-based intelligent QoS optimization algorithm for efficient routing in vehicular networks

With the rapid development of Telematics, Vehicle Self-Organizing Networks (VANETs) play an increasingly critical role in Intelligent Transportation Systems (ITS). Especially in the environment without roadside assistance units (RSUs), how to effectively manage inter-vehicle communication and improve the stability and communication efficiency of the network has become a hot topic of current research. In this paper, a Deep Reinforcement Learning-based Intelligent QoS-optimized efficient routing algorithm for vehicular networks (DRLIQ) is proposed for VANETs with/without RSU environments, and routing methods are proposed respectively. Among them, in RSU-free environment, the DRLIQ algorithm utilizes the powerful processing capability of deep reinforcement learning to intelligently select the optimal data transmission path by dynamically learning and adapting to the changes in the vehicular network, thus effectively reducing communication interruptions and delays, and improving the accuracy of data transmission. The performance of the DRLIQ algorithm under different vehicle densities is evaluated in simulation experiments and compared with current popular algorithms. The experimental results show that the DRLIQ algorithm outperforms the comparison algorithms in reducing the number of communication interruptions, BER and network delay, especially in vehicle-dense environments. In addition, the DRLIQ algorithm shows higher adaptability and stability in coping with network topology changes and vehicle dynamics.

A dual-critic DQN architecture for lifter assignment in multi-floor semiconductor FAB

This paper proposes a deep reinforcement learning-based lifter assignment algorithm utilising Deep Q-network (DQN) to minimise the total inter-floor delivery time in semiconductor manufacturing. Given the complexities and randomness inherent in manufacturing environments, predicting delivery times poses a significant challenge for companies operating in such domains. To address this challenge and improve the accuracy of delay prediction, we partition the end-to-end delivery process of individual lot systematically, focusing on predicting segment delays rather than the overall end-to-end delay. We introduce a unique dual critic architecture designed to handle these segmented steps. This innovative approach enhances accuracy by capturing nuanced information at each step, which is stored as trajectories. Simulation results substantiate the effectiveness of the proposed architecture, comparing favorably against existing algorithms. We conduct comparative analyses with benchmark algorithms, revealing that the proposed algorithm outperforms other algorithms.

Dependent Task Offloading and Resource Allocation via Deep Reinforcement Learning for Extended Reality in Mobile Edge Networks

Extended reality (XR) is an immersive technology widely applied in various fields. Due to the real-time interaction required between users and virtual environments, XR applications are highly sensitive to latency. Furthermore, handling computationally intensive tasks on wireless XR devices leads to energy consumption, which is a critical performance constraint for XR applications. It has been noted that the XR task can be decoupled to several subtasks with mixed serial–parallel relationships. Furthermore, the evaluation of XR application performance involves both subjective assessments from users and objective evaluations, such as of energy consumption. Therefore, in edge computing environments, ways to integrate task offloading for XR subtasks to meet users’ demands for XR applications is a complex and challenging issue. To address this issue, this paper constructs a wireless XR system based on mobile edge computing (MEC) and conducts research on the joint optimization of multi-user communication channel access and task offloading. Specifically, we consider the migration of partitioned XR tasks to MEC servers and formulate a joint optimization problem for communication channel access and task offloading. The objective is to maximize the ratio of quality of experience (QoE) to energy consumption while meeting the user QoE requirements. Subsequently, we introduce a deep reinforcement learning-based algorithm to address this optimization problem. The simulation results demonstrate the effectiveness of this algorithm in meeting user QoE demands and improving energy conversion efficiency, regardless of the XR task partitioning strategies employed.

Energy-Aware Hierarchical Reinforcement Learning Based on the Predictive Energy Consumption Algorithm for Search and Rescue Aerial Robots in Unknown Environments

Aerial robots (drones) offer critical advantages in missions where human participation is impeded due to hazardous conditions. Among these, search and rescue missions in disaster-stricken areas are particularly challenging due to the dynamic and unpredictable nature of the environment, often compounded by the lack of reliable environmental models and limited ground system communication. In such scenarios, autonomous aerial robots’ operation becomes essential. This paper introduces a novel hierarchical reinforcement learning-based algorithm to address the critical limitation of the aerial robot’s battery life. Central to our approach is the integration of a long short-term memory (LSTM) model, designed for precise battery consumption prediction. This model is incorporated into our HRL framework, empowering a high-level controller to set feasible and energy-efficient goals for a low-level controller. By optimizing battery usage, our algorithm enhances the aerial robot’s ability to deliver rescue packs to multiple survivors without the frequent need for recharging. Furthermore, we augment our HRL approach with hindsight experience replay at the low level to improve its sample efficiency.

Interpersonal Communication Interconnection in Media Convergence Metaverse

The metaverse aims to provide immersive virtual worlds connecting with the physical world. To enable real-time interpersonal communications between users across the globe, the metaverse places high demands on network performance, including low latency, high bandwidth, and fast network speeds. This paper proposes a novel Media Convergence Metaverse Network (MCMN) framework to address these challenges. Specifically, the META controller serves as MCMN's logically centralized control plane, responsible for holistic orchestration across edge sites and end-to-end path computation between metaverse users. We develop a model-free deep reinforcement learning-based metaverse traffic optimization algorithm that learns to route flows while satisfying the Quality of Service (QoS) boundaries. The network slicing engine leverages artificial intelligence and machine learning to create isolated, customized virtual networks tailored for metaverse traffic dynamics on demand. It employs unsupervised and reinforcement learning techniques using network telemetry from the META controller to understand application traffic patterns and train cognitive slicer agents to make quality of service -aware decisions accordingly. Optimized delivery of diverse concurrent media types necessitates routing intelligence to meet distinct requirements while mitigating clashes over a shared infrastructure. Media-aware routing enhances traditional shortest-path approaches by combining topological metrics with workflow sensitivities. We realize an edge-assisted rendering fabric to offload complex processing from bandwidth-constrained endpoints while retaining visual realism. Extensive simulations demonstrate MCMN's superior performance compared to conventional networking paradigms. MCMN shows great promise to enable seamless interconnectivity and ultra-high fidelity communications to unlock the true potential of the metaverse.

An IOV Spectrum Sharing Approach based on Multi-Agent Deep Reinforcement Learning

Highly dynamic Internet of Vehicles spectrum sharing can share spectrum owned by vehicle-to-infrastructure links through multiple workshop links to achieve efficient resource allocation. Aiming at the problem that the rapid variations in channel states in highly dynamic vehicular environments can make it challenging for base stations to gather and manage information about instantaneous channel states, we present a multi-agent deep reinforcement learning-based V2X spectrum access algorithm. The algorithm is designed to optimize the throughput of V2I user under V2V user delay and reliability constraints, and uses the experience gained from interacting with the communication environment to update the Q network to improve spectrum and power allocation strategies. Implicit collaborative agents are trained through an improved DQN model combined with dueling network architecture and long short-term memory network layers and public rewards. With lagged Q-learning and concurrent experience replay trajectories, the training process was stabilized and the non-stationarity problem caused by concurrent learning of multiple agents was resolved. Simulation results demonstrate that our presented algorithm achieves a mean successful payload delivery rate of 95.89%, which is 16.48% greater than that of the randomized baseline algorithm. Our algorithm obtains approximately the optimal value and shows performance close to the centralized brute force algorithm, which provides a better strategy for further minimizing the signaling overhead of the Internet of Vehicles communication system.

Inverse kinematics solution and control method of 6-degree-of-freedom manipulator based on deep reinforcement learning

The advent of Industry 4.0 has significantly promoted the field of intelligent manufacturing, which is facilitated by the development of new technologies are emerging. Robot technology and robot intelligence methods have rapidly developed and been widely applied. Manipulators are widely used in industry, and their control is a crucial research topic. The inverse kinematics solution of manipulators is an important part of manipulator control, which calculates the joint angles required for the end effector to reach a desired position and posture. Traditional inverse kinematics solution algorithms often face the problem of insufficient generalization, and iterative methods have challenges such as large computation and long solution time. This paper proposes a reinforcement learning-based inverse kinematics solution algorithm, called the MAPPO-IK algorithm. The algorithm trains the manipulator agent using the MAPPO algorithm and calculates the difference between the end effector state of the manipulator and the target posture in real-time by designing a reward mechanism, while considering Gaussian distance and cosine distance. Through experimental comparative analysis, the feasibility, computational efficiency, and superiority of this reinforcement learning algorithm are verified. Compared with traditional inverse kinematics solution algorithms, this method has good generalization and supports real-time computation, and the obtained result is a unique solution. Reinforcement learning algorithms have better adaptability to complex environments and can handle different sudden situations in different environments. This algorithm also has the advantages of path planning, intelligent obstacle avoidance, and other advantages in dynamically processing complex environmental scenes.

Classification with costly features in hierarchical deep sets

Classification with costly features (CwCF) is a classification problem that includes the cost of features in the optimization criteria. Individually for each sample, its features are sequentially acquired to maximize accuracy while minimizing the acquired features’ cost. However, existing approaches can only process data that can be expressed as vectors of fixed length. In real life, the data often possesses rich and complex structure, which can be more precisely described with formats such as XML or JSON. The data is hierarchical and often contains nested lists of objects. In this work, we extend an existing deep reinforcement learning-based algorithm with hierarchical deep sets and hierarchical softmax, so that it can directly process this data. The extended method has greater control over which features it can acquire and, in experiments with seven datasets, we show that this leads to superior performance. To showcase the real usage of the new method, we apply it to a real-life problem of classifying malicious web domains, using an online service.

Reinforcement Learning-Based Multiobjective Evolutionary Algorithm for Mixed-Model Multimanned Assembly Line Balancing Under Uncertain Demand.

In practical assembly enterprises, customization and rush orders lead to an uncertain demand environment. This situation requires managers and researchers to configure an assembly line that increases production efficiency and robustness. Hence, this work addresses cost-oriented mixed-model multimanned assembly line balancing under uncertain demand, and presents a new robust mixed-integer linear programming model to minimize the production and penalty costs simultaneously. In addition, a reinforcement learning-based multiobjective evolutionary algorithm (MOEA) is designed to tackle the problem. The algorithm includes a priority-based solution representation and a new task-worker-sequence decoding that considers robustness processing and idle time reductions. Five crossover and three mutation operators are proposed. The Q -learning-based strategy determines the crossover and mutation operator at each iteration to effectively obtain Pareto sets of solutions. Finally, a time-based probability-adaptive strategy is designed to effectively coordinate the crossover and mutation operators. The experimental study, based on 269 benchmark instances, demonstrates that the proposal outperforms 11 competitive MOEAs and a previous single-objective approach to the problem. The managerial insights from the results as well as the limitations of the algorithm are also highlighted.

Deep Reinforcement Learning based channel allocation (DRLCA) in Cognitive Radio Networks

The Spectrum is a scarce resource in wireless networks, hence these resources need to be perfectly channelized for their better utilization. By the study of the previous decades, it is found that the spectrum is colossally underutilized and the main reason for the underutilization found out to be the policies that are fixed and not dynamic. The dynamic spectrum allocation of frequency bands may overcome this problem. Cognitive radio provides an important concept that can used to solve the problem of underutilization of spectrum. Reinforcement learning is a key technique that is widely used to learn the spectrum allocation behaviour and maximize the system’s efficiency. Therefore, in this work, we have designed and developed a Reinforcement learning-based model to allocate the channels among secondary users. Also, a Deep Reinforcement learning-based channel allocation algorithm (DRLCA) has been proposed. The proposed DRLCA is compared with existing JPCRL [1]. In our algorithm, the Python libraries were used for simulation. From the simulation results and analysis, it is found that the DRLCA outperforms the JPCRL in terms of channel utilization by 5%.

A Reinforcement Learning-Based Traffic Engineering Algorithm for Enterprise Network Backbone Links

Large enterprise networks typically rely on expensive, high-speed backbone links to connect multiple campuses across diverse regions. As the volume of traffic traversing these backbone links increases, traffic engineering techniques are employed to filter or redirect traffic flows. Nevertheless, simple rerouting strategies can introduce business disruptions such as packet reordering, which significantly impact the user experience. To address this issue, we introduce an enhanced traffic scheduling algorithm named Critical Flow Rerouting with Weight- Reinforcement Learning(CFRW-RL), which builds upon the critical flow rerouting-reinforcement learning (CFR-RL) algorithm. CFRW-RL incorporates the principles of reinforcement learning, accounting for both the weights and classifications of data flows. This approach enables the algorithm to prioritize flows with lower weights for rerouting. The simulation results demonstrate that CFRW-RL significantly minimizes the rerouting of high-priority business flows and reduces business interference compared with the CFR-RL algorithm and that it maintains a similar computational complexity.

Machine Learning-Based Beamforming Algorithm for Massive MIMO Systems in 5G Networks

The following research focuses on the use of machine learning-based beamforming algorithms to improve Massive Multiple Input Multiple Output (MIMO) systems in 5G networks. Four unique algorithms namely, the Deep Learning Beamforming Algorithm (DLBA), Reinforcement Learning-Based Doa Estimation Algorithm (RLBEA), Clustering based beam forming algorithm(CBA) and GeneticAlgorithm Based Beam Forming Algoeithm were developed after which each of them was undertook evaluation. Widespread trials, in a simulated 5G environment, have revealed that the DLBA and RLBA considerably outperform other technologies by means of system throughput SINR as well Both the DLBA and RLBA achieved high system throughput, increased SINR levels and low BER. CBA and GABA, using clustering and genetic algorithms as their approaches, displayed moderate values on all assessed composite measures. This research offers important insights on the adaptability and learning potential of machine-learning based beamforming algorithms highlighting their ability to improve efficiency in wireless communication networks during the 5G revolution

Learning Audio and Video Bitrate Selection Strategies via Explicit Requirements

Mobile video streaming dominates today's network traffic, and adaptive bitrate (ABR) algorithms have been routinely adopted for transmitting media content across dynamic mobile networks. State-of-the-art ABR algorithms mainly alter video bitrate without considering audio bitrate as they consider the impact on the video negligible due to their small size. However, to bring users an immersive experience, recent content providers have applied high-quality audio with large sizes, like stereophonic sound. Therefore, improper audio bitrate selection will adversely affect video bitrate selection, leading to undesirable audio/video combinations (the highest video quality with the lowest audio quality, and vice versa) and frequent playback interruptions. To address these inefficiencies, we propose a Self-Play reinforcement learning-based Audio-aware ABR algorithm named SPA to learn strategies for audio and video bitrate selections. By learning from explicit goals, SPA can match the actual requirements and attain good performance. By conducting trace-driven and testbed-based experiments, we observe SPA's considerable superiority compared to existing approaches, including reducing the undesirable combinations by up to 34.17× and achieving zero stall time across 88.57% of traces. We also invite 35 volunteers to join a subjective test, and the result shows that 33/35 people consider SPA provides them with a satisfactory viewing experience.

Research on collaborative edge network service migration strategy based on crowd clustering

The innovative application of Crowd Intelligent Devices (CIDS) in edge networks has garnered attention due to the rapid development of artificial intelligence and computer technology. This application offers users more reliable and low-latency computing services through computation offloading technology. However, the dynamic nature of network terminals and the limited coverage of edge servers pose challenges, such as data loss and service interruption. Furthermore, the high-speed mobility of intelligent terminals in the dynamic edge network environment further complicates the design of computation offloading and service migration strategies. To address these challenges, this paper explores the computation offloading model of cluster intelligence collaboration in a heterogeneous network environment. This model involves multiple intelligences collaborating to provide computation offloading services for terminals. To accommodate various roles, a switching strategy of split-cluster group collaboration is introduced, assigning the cluster head, the alternate cluster head, and the ordinary user are assigned to a group with different functions. Additionally, the paper formulates the optimal offloading strategy for group smart terminals as a Markov decision process, taking into account factors such as user mobility, service delay, service accuracy, and migration cost. To implement this strategy, the paper utilizes the deep reinforcement learning-based CCSMS algorithm. Simulation results demonstrate that the proposed edge network service migration strategy, rooted in groupwise cluster collaboration, effectively mitigates interruption delay and enhances service migration efficiency.

Boundary Optimal Control for Parabolic Distributed Parameter Systems With Value Iteration.

A reinforcement learning-based boundary optimal control algorithm for parabolic distributed parameter systems is developed in this article. First, a spatial Riccati-like equation and an integral optimal controller are derived in infinite-time horizon based on the principle of the variational method, which avoids the complex semigroups and operator theories. Using state data along the system trajectory, a value iteration algorithm via the Bellman optimality principle is proposed to obtain the solution of the spatial Riccati-like equation and the optimal control law. The convergence of the value iteration algorithm is proved. Subsequently, an approximation scheme based on weighted residuals is developed to implement the value iteration algorithm, where radial basis functions are chosen as the basic functions to approximate the solution of the spatial Riccati-like equation. Simulations on the diffusion-reaction process demonstrate the effectiveness of the developed method.

Carbon emission causal discovery and multi-step forecasting for global cities

The increasing threat of global climate change is primarily caused by rising carbon emissions, with cities acting as significant contributors. This study bridges two vital gaps in urban carbon neutrality research: unraveling the causal dynamics of carbon emissions within urban networks and forecasting emission trends. This study proposes a reinforcement learning-based causal discovery algorithm, progressively deciphering the complex causal relationships in global urban emissions, and facilitating the creation of directed acyclic causal graphs. Furthermore, a hyperbolic graph neural network-based forecasting algorithm is introduced, through integrated fusion curvature to improve the information transfer between cities, for predicting global urban emission trends. A comparative analysis positions these innovative algorithms against leading methods, using emission data from thousands of cities for predictions one, five, and ten steps ahead. The experiment employs prediction error metrics, Taylor statistics, the Diebold-Mariano test, and the ablation analysis for validation. Results reveal proposed causal discovery algorithm effectively identifies the causality of carbon emissions among cities, while the forecasting algorithm leads other competing models across all prediction ranges. Based on the effectiveness of the algorithms, this study decodes the significant nature of the global urban carbon emission network, offering policy insights for collaborative carbon mitigation in cities worldwide.

Deep reinforcement learning-based memetic algorithm for energy-aware flexible job shop scheduling with multi-AGV

The integration of manufacturing and logistics scheduling issues in shop operations has garnered considerable attention. Concurrently, escalating concerns about global warming have propelled the emergence of green manufacturing as a critical challenge. Notably, extant research in this domain lacks an incorporation of green metrics within the framework of manufacturing and logistics-integrated scheduling. Furthermore, the determination of a critical block remains a challenging aspect, with an absence of consideration for a neighborhood structure founded on the critical block. Moreover, prior endeavors have predominantly relied on Q-learning to augment evolutionary algorithms, a strategy criticized for its limited learning capacity. Consequently, this study addresses these gaps by presenting an energy-efficient flexible job Shop scheduling with multi-autonomous guided vehicles (EFJS-AGV). The primary objectives are the simultaneous minimization of makespan and total energy consumption. To tackle this NP-hard problem, a deep Q-network-based memetic algorithm is proposed. The devised algorithm incorporates several distinctive features. Firstly, the strength Pareto evolutionary algorithm (SPEA2) is employed to swiftly explore the objective space, enhancing convergence and diversity. Secondly, four distinct local search operators based on critical paths and blocks are devised to efficiently reduce makespan. Thirdly, deep reinforcement learning is harnessed to understand the interplay between solutions and action selection. This understanding aids the evolutionary algorithm in selecting the most optimal operator. The efficacy of the proposed algorithm is rigorously evaluated through a comparative analysis with five state-of-the-art algorithms. The assessment is conducted on two benchmark datasets encompassing 20 instances. The numerical experimental results affirm the effectiveness of the proposed enhancements and algorithms. Furthermore, the superior performance of the proposed algorithm in addressing the EFJS-AGV substantiates its robustness and applicability.

Reinforcement learning-based multi-objective differential evolution algorithm for feature selection

Feature Selection (FS) can be used to determine the optimal subset of features from a raw dataset by reducing dimensionality and improving accuracy. In this study, a reinforcement learning-based multi-objective differential evolution algorithm (RLMODE) for FS is proposed, which is modeled as a multi-objective optimization problem. First, a reinforcement learning-based offspring generation strategy is designed. The offspring generation strategy is based on the Q-learning framework, which considers each individual in the population as an agent. The dominance relationship between an agent and its predecessor is used to encode the state. A well-chosen action set containing three typical differential evolution mutation operators is available for each agent. The reward is used to update the exclusive Q-table. Moreover, a novel Pareto front (PF) relearning strategy is devised to allow adequate communication between individuals on the PF. The PF relearning strategy reevaluates the potential value of all individuals on the PF as a whole. This promotes the propagation of excellent solutions and improves PF diversity. The proposed RLMODE initially demonstrates its strength in benchmarks. The excellent results for 17 datasets demonstrate RLMODE’s merits in terms of dimensionality reduction and accuracy improvement. Therefore, the proposed RLMODE method is a promising FS technique.