Reinforcement Learning Algorithm Research Articles

Optimizing Microservice Deployment in Edge Computing with Large Language Models: Integrating Retrieval Augmented Generation and Chain of Thought Techniques

Large Language Models (LLMs) have demonstrated impressive capabilities in autogenerating code based on natural language instructions provided by humans. We observed that in the microservice models of edge computing, the problem of deployment latency optimization can be transformed into an NP-hard mathematical optimization problem. However, in the real world, deployment strategies at the edge often require immediate updates, while human-engineered code tends to be lagging. To bridge this gap, we innovatively integrated LLMs into the decision-making process for microservice deployment. Initially, we constructed a private Retrieval Augmented Generation (RAG) database containing prior knowledge. Subsequently, we employed meticulously designed step-by-step inductive instructions and used the chain of thought (CoT) technique to enable the LLM to learn, reason, reflect, and regenerate. We decomposed the microservice deployment latency optimization problem into a collection of granular sub-problems (described in natural language), progressively providing instructions to the fine-tuned LLM to generate corresponding code blocks. The generated code blocks underwent integration and consistency assessment. Additionally, we prompted the LLM to generate code without the use of the RAG database for comparative analysis. We executed the aforementioned code and comparison algorithm under identical operational environments and simulation parameters, conducting rigorous result analysis. Our fine-tuned model significantly reduced latencies by 22.8% in handling surges in request flows, 37.8% in managing complex microservice types, and 39.5% in processing increased network nodes compared to traditional algorithms. Moreover, our approach demonstrated marked improvements in latency performance over LLMs not utilizing RAG technology and reinforcement learning algorithms reported in other literature. The use of LLMs also highlights the concept of symmetry, as the symmetrical structure of input-output relationships in microservice deployment models aligns with the LLM’s inherent ability to process and generate balanced and optimized code. Symmetry in this context allows for more efficient resource allocation and reduces redundant operations, further enhancing the model’s effectiveness. We believe that LLMs hold substantial potential in optimizing microservice deployment models.

Retraction Note: Analysis on quantum reinforcement learning algorithms for prediction of protein sequence

Retraction Note: Analysis on quantum reinforcement learning algorithms for prediction of protein sequence

Decision-making optimization for post-disaster restoration of multimodal transport networks in terms of resilience

IntroductionPublic health emergencies can have a ripple effect on the resilience of multimodal transport network, which will lead to problems such as route disruptions or blockages, route selection change and information transmission delay spreading to the whole network, further hindering the transportation planning and operational efficiency of the network. MethodsThis study constructs a multimodal transport route optimization model under uncertainty with the objective of the sum of transportation cost, transshipment cost, penalty cost and carbon emission cost. To enhance the computational efficiency of the model, a novel invasive weed optimization with memory and encoding value clustering capabilities is proposed. In addition, by fusing the Q-learning algorithm in reinforcement learning with the novel invasive weed algorithm, the action-value function table obtained from the training facilitates the selection of optimal routes. Based on empirical data, explore the sensitivity analysis of node disruptions, time windows, and fuzzy demand on route decision-making under public health emergencies. ResultsThe transport network is affected by public health emergencies, which makes the optimal route deviate from the expected goal, resulting in an increase in the total cost. The proportion of total cost is determined by the position of nodes in the network, with critical nodes suffering more losses than ordinary nodes. Reasonable setting of time windows and fuzzy demand intervals is an effective way to improve the resilience and transportation efficiency of multimodal transport network. ConclusionsThis study provides more applicable decision-making references for enterprises to prevent the risk of supply chain disruptions caused by public health emergencies.

Research on Control Strategy of Hybrid Superconducting Energy Storage Based on Reinforcement Learning Algorithm

Research on Control Strategy of Hybrid Superconducting Energy Storage Based on Reinforcement Learning Algorithm

A Model-Based Reinforcement Learning Algorithm for Multi-Agent Cooperation Nash Equilibrium With Unstable Communication

Solving Nash equilibrium is important to multi-agent systems, in which the communication is an important factor. However, many proposed reinforcement learning (RL) based algorithms take into account the communication factors by assuming stable communication conditions, which does not hold in the real environment. In this paper, we analyze the effect of a typical RL algorithm in the cases of unstable communication and communication failure, which causes information loss between agents, leading to isolation of agents and affecting algorithm convergence. Then, we propose a model-based RL algorithm to solve Nash equilibrium for multi-agent systems when agents are isolated, and prove its convergence and rationality through mathematical proofs. The simulations results show the effectiveness of the proposed algorithm.

Semi-Supervised Intrusion Detection System for In-Vehicle Networks Based on Variational Autoencoder and Adversarial Reinforcement Learning

Despite the affordability, simplicity, and efficiency of controller area network (CAN) protocols, the security vulnerability remains a major challenge. Currently, a machine learning-based intrusion detection system (IDS) is considered an effective approach for improving security in CAN by identifying malicious attacks. However, earlier studies that relied on supervised learning methods required considerable amounts of labeled data. Data collection from vehicles is time-consuming and expensive. Furthermore, the obtained data exhibited a class imbalance, which presents further challenges in the analysis and model training. Thus, we propose a semi-supervised learning-based IDS that combines variational autoencoder (VAE) and adversarial reinforcement learning for the multi-class classification of both known and unknown attacks. The proposed system capitalizes on the diverse patterns inherent in unlabeled data, transforming this data space into one that is more conducive to classification. Concurrently, adversarial agents in the reinforcement learning algorithm interact competitively, progressively enhancing their ability to intelligently classify and select samples. To reduce the reliance on labeled data and effectively exploit them, we utilize a pseudo-labeling process for pre-training. Experimental results indicate that the proposed model achieves more effective classification while requiring less labeled data compared to other baseline models for known attacks. By inheriting the advantages of VAE, promising results demonstrate that the proposed system detects unknown attacks containing similar or completely different characteristics with high F1 scores exceeding 0.9 and 0.84, respectively. Finally, the proposed system was demonstrated to be a lightweight model for the expeditious detection of malevolent messages introduced into in-vehicle networks to ensure minimal latency.

Probing an LSTM-PPO-Based reinforcement learning algorithm to solve dynamic job shop scheduling problem

With the growth of personalized demand and the continuous improvement in social productivity, the large-scale and few-variety centralized production model is gradually transitioning towards a personalized model of small batches and multiple varieties, which makes the manufacturing process of the job shop increasingly complex. Furthermore, disruptive events such as machinery failures and rush orders in the job shop increase the uncertainty and variability of the production environment. Traditional scheduling methods are usually based on fixed rules and heuristic algorithms, which are difficult to adapt to constantly changing production environments and demands. This may lead to inaccurate scheduling decisions and hinder the optimal allocation of job shop resources. To solve the dynamic job shop scheduling problem (JSP) more effectively, this paper proposes a Reinforcement Learning (RL) optimization algorithm integrating long short-term memory (LSTM) neural network and proximal policy optimization (PPO). It can dynamically adjust scheduling strategies according to the changing production environment, achieving comprehensive status awareness of the job shop environment to make optimal scheduling decisions. First, a state-aware network framework based on LSTM-PPO is proposed to achieve real-time perception of job shop state changes. Then, the state and action space of the job shop are described within the context of the state-aware network framework. Finally, an experimental environment is established to verify the algorithm’s effectiveness. Training the LSTM-PPO algorithm makes it feasible to achieve better performance than other scheduling methods. By comparing the initial planning time with the actual completion time of the rescheduling decision under different dynamic disturbances, the efficiency of the proposed algorithm is verified for the dynamic JSP

PSOR03 Presentation Time: 11:40 AM: Reinforcement Learning Algorithm for Catheter Optimization in HDR Prostate Brachytherapy

PSOR03 Presentation Time: 11:40 AM: Reinforcement Learning Algorithm for Catheter Optimization in HDR Prostate Brachytherapy

Reinforcement Learning for Synchronization of Heterogeneous Multiagent Systems by Improved Q-Functions.

This article dedicates to investigating a methodology for enhancing adaptability to environmental changes of reinforcement learning (RL) techniques with data efficiency, by which a joint control protocol is learned using only data for multiagent systems (MASs). Thus, all followers are able to synchronize themselves with the leader and minimize their individual performance. To this end, an optimal synchronization problem of heterogeneous MASs is first formulated, and then an arbitration RL mechanism is developed for well addressing key challenges faced by the current RL techniques, that is, insufficient data and environmental changes. In the developed mechanism, an improved Q-function with an arbitration factor is designed for accommodating the fact that control protocols tend to be made by historic experiences and instinctive decision-making, such that the degree of control over agents' behaviors can be adaptively allocated by on-policy and off-policy RL techniques for the optimal multiagent synchronization problem. Finally, an arbitration RL algorithm with critic-only neural networks is proposed, and theoretical analysis and proofs of synchronization and performance optimality are provided. Simulation results verify the effectiveness of the proposed method.

A Test Report Optimization Method Fusing Reinforcement Learning and Genetic Algorithms

Filtering high-variability and high-severity defect reports from large test report databases is a challenging task in crowdtesting. Traditional optimization algorithms based on clustering and distance techniques have made progress but are limited by initial parameter settings and significantly decrease in efficiency with an increasing number of reports. To address this issue, this paper proposes a method that integrates reinforcement learning with genetic algorithms for crowdsourced testing report optimization, called Reinforcement Learning-based Genetic Algorithm for Crowdsourced Testing Report Optimization (RLGA). Its core goal is to identify distinct, high-severity defect reports from a large set. The method uses genetic algorithms to generate the optimal report selection sequence and adjusts the crossover probability (Pc) and mutation probability (Pm) dynamically with reinforcement learning based on the population’s average fitness, best fitness, and diversity. The reinforcement learning component uses a hybrid SARSA and Q-Learning strategy to update the Q-value table, allowing the algorithm to learn quickly in early iterations and expand the search space later to avoid local optima, thereby improving efficiency. To validate the RLGA method, this paper uses four public datasets and compares RLGA with six classic methods. The results indicate that RLGA outperforms BDDIV in terms of execution time and is less sensitive to the total number of test reports. In terms of optimization objectives, the test reports selected by RLGA have higher levels of defect severity and diversity than those selected by the random choice, BDDIV, and TSE methods. Regarding population diversity, RLGA effectively enhances the uniformity and diversity of individuals compared to random initialization. In terms of convergence speed, RLGA is superior to the GA, GA-SARSA, and GA-Q methods.

Human-like Decision Making for autonomous lane change driving: a hybrid inverse reinforcement learning with game-theoretical vehicle interaction model

The development of automated driving vehicles aims to provide safer, comfortable, and more efficient mobility options. However, the decision-making control of autonomous vehicles still embraces limitations on human performance mimicry. These limitations become particularly evident in complex and unfamiliar driving scenarios, where weak decisionmaking abilities and poor adaptation of vehicle behaviour are prominent issues. This paper proposes a game-theoretic decisionmaking algorithm for human-like driving in the vehicle lane change scenario. Firstly, an inverse reinforcement learning (IRL) model is used to quantitatively analyse the lane change trajectories of the natural driving dataset, establishing the human-like human cost function. Subsequently, joint safety, comfort to build the comprehensive decision cost function. Use the combined decision cost function to conduct a non-cooperative game of vehicle lane changing decision to solve the optimal decision of host vehicle lane changing. The host vehicle lane-changing decision problem is formulated as a Stackelberg game optimization problem. To verify the feasibility and effectiveness of the algorithm proposed in this study, a lane change test scenario has been established. Firstly, we analyse the human-like decision-making model derived by the maximum entropy inverse reinforcement learning algorithm to verify the effectiveness and robustness of the IRL algorithm. Secondly, the human-like game decisionmaking algorithm in this paper is validated by conducting an interactive lane-changing experiment with obstacle vehicles of different driving styles. The experimental results prove that the human-like driving decision-making model proposed in this study can make lane-changing behaviours in line with human driving patterns in lane-changing scenarios of expressway.

Deep Reinforcement Learning for UAV-Based SDWSN Data Collection

Recent advancements in Unmanned Aerial Vehicle (UAV) technology have made them effective platforms for data capture in applications like environmental monitoring. UAVs, acting as mobile data ferries, can significantly improve ground network performance by involving ground network representatives in data collection. These representatives communicate opportunistically with accessible UAVs. Emerging technologies such as Software Defined Wireless Sensor Networks (SDWSN), wherein the role/function of sensor nodes is defined via software, can offer a flexible operation for UAV data-gathering approaches. In this paper, we introduce the “UAV Fuzzy Travel Path”, a novel approach that utilizes Deep Reinforcement Learning (DRL) algorithms, which is a subfield of machine learning, for optimal UAV trajectory planning. The approach also involves the integration between UAV and SDWSN wherein nodes acting as gateways (GWs) receive data from the flexibly formulated group members via software definition. A UAV is then dispatched to capture data from GWs along a planned trajectory within a fuzzy span. Our dual objectives are to minimize the total energy consumption of the UAV system during each data collection round and to enhance the communication bit rate on the UAV-Ground connectivity. We formulate this problem as a constrained combinatorial optimization problem, jointly planning the UAV path with improved communication performance. To tackle the NP-hard nature of this problem, we propose a novel DRL technique based on Deep Q-Learning. By learning from UAV path policy experiences, our approach efficiently reduces energy consumption while maximizing packet delivery.

Resource allocation optimization of device-to-device communication using machine learning algorithms

Device-to-device communication is envisioning next-generation wireless communication. The utility of the device-to-device communication model encourages emerging communication systems such as 5G and the Internet of Things. The allocation of resources and channel interference are major challenges in device-to-device communication. This paper proposes machine-learning-based algorithms for resource allocation and optimization of device-to-device communication. The proposed machine learning algorithm is a cascaded support vector machine. The cascaded support vector machine mapped the parameters of CUEs and DUEs. We create an iterative algorithm to achieve low-power, energy-efficient resource allocation with mode selection by formulating a novel optimization problem to maximize energy efficiency using the subtractive form method to solve a fractional objective function. We obtain data samples from a suboptimal algorithm to train the cascaded algorithm and verify the trained algorithm. Our numerical results show that the proposed cascaded machine learning-based transmission algorithm's accuracy reaches about 88%–95% despite its simple structure due to the limitation in computing power. The analysis of results suggests that the proposed algorithm is more efficient than SVM, DSVM, and the reinforcement learning (RL) algorithm.

Machine Learning Based Mechanical Fault Diagnosis and Detection Methods: A Systematic Review

Abstract Mechanical fault diagnosis and detection are crucial for enhancing equipment reliability, economic efficiency, production safety, and energy conservation. In the era of Industry 4.0, artificial intelligence (AI) has emerged as a significant tool for mechanical fault diagnosis and detection, attracting considerable attention from both academia and industry. This review focuses on the application of AI techniques in mechanical fault diagnosis and detection using artificial intelligence techniques based on the existing research. It examines various AI algorithms including k-nearest neighbors, support vector machine, artificial neural network, deep learning, reinforcement learning, computer vision, and transformer algorithm integrating theoretical foundations with practical applications in industrial production. Furthermore, a comprehensive overview of these algorithms applications in mechanical fault diagnosis and detection is provided. Finally, a critical assessment highlights the advantages and limitations of these techniques, while forecasting the developmental trajectories of future intelligent diagnostic technologies based on machine learning. This review serves to bridge the gap between researchers in AI and fault diagnosis, contributing significantly to the field.

AI-Augmented Chemical Engineering Models for Predictive Battery Maintenance in IoT Applications

This research presents a cutting-edge approach that combines chemical engineering principles with artificial intelligence (AI) to enhance battery maintenance in Internet of Things (IoT) applications. The study develops predictive maintenance models capable of accurately forecasting battery degradation, thereby optimizing usage patterns and extending battery life. By integrating electrochemical data with AI-driven predictive analytics, the project enables real-time monitoring of battery health, achieving over 90% accuracy in anticipating maintenance needs, which reduces unplanned downtime and lowers operational costs. Employing machine learning frameworks, data analysis tools, and chemical engineering simulation software, the focus is on lithium-ion and solid-state batteries, commonly used in IoT networks across sectors such as smart cities, agriculture, and healthcare. Key results include a 20−30% increase in battery lifespan and a 15% improvement in energy efficiency, significantly reducing electronic waste and environmental footprint. This interdisciplinary approach provides a foundation for sustainable battery technology in IoT technology, paving the way for future advancements in AI-augmented predictive maintenance and chemical engineering models for next-generation applications. The integration of AI with chemical engineering principles allows for the development of models that adapt to the unique electrochemical behavior of different battery types, such as lithium-ion and solid-state batteries. By analyzing lifecycle data and device usage patterns, predictive models can account for variables that influence battery performance over time, including temperature fluctuations and charging cycles. This project utilizes neural networks and reinforcement learning algorithms to train models capable of learning from historical data, enhancing their predictive capabilities. A significant aspect of this research involves the preprocessing of battery lifecycle data to ensure high data quality, which is critical for accurate model training and validation.

Retraction Note: A full freedom pose measurement method for industrial robot based on reinforcement learning algorithm

Retraction Note: A full freedom pose measurement method for industrial robot based on reinforcement learning algorithm

Reinforcement Learning Algorithm for Optimising Durian Irrigation Systems: Maximising Growth and Water Efficiency

This study presents a Reinforcement Learning-based algorithm designed to optimise irrigation for Durio Zibethinus (i.e., durian) trees, aiming to maximise tree growth and reduce water usage. Traditional irrigation methods, as well as current machine learning models, often focus only on soil moisture and weather data, neglecting critical factors like actual tree growth. This study proposed a reinforcement learning irrigation (RL-Irr) algorithm incorporating tree growth stages, soil moisture, and weather conditions to determine precise irrigation needs. The algorithm was developed by calibrating the AQUACROP model using data from actual durian plantations where rain-fed irrigation (rain-fed) was practised. Daily irrigation volumes were calculated based on real-time soil moisture, weather forecasts, and weekly tree growth measurements. The reinforcement learning method was used to optimise irrigation schedules, with rewards based on soil moisture, tree growth, rainfall, and weather conditions. The algorithm was tested using AQUACROP simulations and compared against soil moisture balance irrigation (SMB-Irr) and rain-fed. The results showed that the RL-Irr reduced water use by up to 75 percent while maintaining tree growth. These findings suggest the algorithm could significantly improve water efficiency in durian farming, though real-world applications should consider potential model limitations.

Short-Term Electricity Futures Investment Strategies for Power Producers Based on Multi-Agent Deep Reinforcement Learning

The global development and enhancement of electricity financial markets aim to mitigate price risk in the electricity spot market. Power producers utilize financial derivatives for both hedging and speculation, necessitating careful selection of portfolio strategies. Current research on investment strategies for power financial derivatives primarily emphasizes risk management, resulting in a lack of a comprehensive investment framework. This study analyzes six short-term electricity futures contracts: base day, base week, base weekend, peak day, peak week, and peak weekend. A multi-agent deep reinforcement learning algorithm, Dual-Q MADDPG, is employed to learn from interactions with both the spot and futures market environments, considering the hedging and speculative behaviors of power producers. Upon completion of model training, the algorithm enables power producers to derive optimal portfolio strategies. Numerical experiments conducted in the Nordic electricity spot and futures markets indicate that the proposed Dual-Q MADDPG algorithm effectively reduces price risk in the spot market while generating substantial speculative returns. This study contributes to lowering barriers for power generators in the power finance market, thereby facilitating the widespread adoption of financial instruments, which enhances market liquidity and stability.

Online Three-Dimensional Fuzzy Reinforcement Learning Modeling for Nonlinear Distributed Parameter Systems

Distributed parameter systems (DPSs) frequently appear in industrial manufacturing processes, with complex characteristics such as time–space coupling, nonlinearity, infinite dimension, uncertainty and so on, which is full of challenges to the modeling of the system. At present, most DPS modeling methods are offline. When the internal parameters or external environment of DPS change, the offline model is incapable of accurately representing the dynamic attributes of the real system. Establishing an online model for DPS that accurately reflects the real-time dynamics of the system is very important. In this paper, the idea of reinforcement learning is creatively integrated into the three-dimensional (3D) fuzzy model and a reinforcement learning-based 3D fuzzy modeling method is proposed. The agent improves the strategy by continuously interacting with the environment, so that the 3D fuzzy model can adaptively establish the online model from scratch. Specifically, this paper combines the deterministic strategy gradient reinforcement learning algorithm based on an actor critic framework with a 3D fuzzy system. The actor function and critic function are represented by two 3D fuzzy systems and the critic function and actor function are updated alternately. The critic function uses a TD (0) target and is updated via the semi-gradient method; the actor function is updated by using the chain derivation rule on the behavior value function and the actor function is the established DPS online model. Since DPS modeling is a continuous problem, this paper proposes a TD (0) target based on average reward, which can effectively realize online modeling. The suggested methodology is implemented on a three-zone rapid thermal chemical vapor deposition reactor system and the simulation results demonstrate the efficacy of the methodology.

Cross-Layer Routing Protocol Based on Channel Quality for Underwater Acoustic Communication Networks

Due to the physical characteristics of acoustic channels, the performance of underwater acoustic communication networks (UACNs) is more susceptible to the impacts of multipath and Doppler effects. Channel quality can serve as a measure of the reliability of underwater communication links. A cross-layer routing protocol based on channel quality (CLCQ) is proposed to improve the overall network performance and resource utilization. First, the BELLHOP ray model is used to calculate the channel impulse response combined with the winter sound speed profile data of a specific sea area. Then, the channel impulse response is integrated into the communication system to evaluate the channel quality between nodes based on the bit error rate (BER). Finally, during the selection of the next hop node, a reinforcement learning algorithm is employed to facilitate cross-layer interaction within the protocol stack. The optimal relay node is determined by the channel quality index (BER) from the physical layer, the buffer state from the data link layer, and the node residual energy. To enhance the algorithm’s convergence speed, a forwarding candidate set selection method is proposed which takes into account node depth, residual energy, and buffer state. Simulation results show that the packet delivery rate (PDR) of the CLCQ is significantly higher than that of Q-Learning-Based Energy-Efficient and Lifetime-Extended Adaptive Routing (QELAR) and Geographic and Opportunistic Routing (GEDAR).