Reinforcement Learning Scheme Research Articles

Combination of fuzzy control and reinforcement learning for wind turbine pitch control

Abstract The generation of the pitch control signal in a wind turbine (WT) is not straightforward due to the nonlinear dynamics of the system and the coupling of its internal variables; in addition, they are subjected to the uncertainty that comes from the random nature of the wind. Fuzzy logic has proved useful in applications with changing system parameters or where uncertainty is relevant as in this one, but the tuning of the fuzzy logic controller (FLC) parameters is neither straightforward nor an easy task. On the other hand, reinforcement learning (RL) allows systems to automatically learn, and this capability can be exploited to tune the FLC. In this work, a WT pitch control architecture that uses RL to tune the membership functions and scale the output of a fuzzy controller is proposed. The RL strategy calculates the fuzzy controller gains in order to reduce the output power error of the WT according to the wind speed. Different reward mechanisms based on the output power error have been considered. Simulation results with different wind profiles show that this architecture performs better (123.7 W) in terms of power errors than an FLC without RL (133.2 W) or a simpler PID (208.8 W). Even more, it provides a smooth response and outperforms other hybrid controllers such as RL-PID and radial basis function neural network control.

Design and Implementation of a Low-Cost Intelligent Unmanned Surface Vehicle

This article describes the design and construction journey of a self-developed unmanned surface vehicle (USV). In order to increase the accessibility and lower the barrier of entry we propose a low-cost (under EUR 1000) approach to the vessel construction with great adaptability and customizability. This design prioritizes minimal power consumption as a key objective. It focuses on elucidating the intricacies of both the design and assembly processes involved in creating an economical USV. Utilizing easily accessible components, the boat outlined in this study has been already participated in the 1st Aegean Ro-boat Race 2023 competition and is tailored for entry into similar robotic competitions. Its primary functionalities encompass autonomous sea navigation coupled with sophisticated collision avoidance capabilities. Finally, we studied reinforcement learning strategies for constructing a robust intelligent controller for the task of USV navigation under disturbances and we show some preliminary simulation results we have obtained.

Application Study on the Reinforcement Learning Strategies in the Network Awareness Risk Perception and Prevention

The intricacy of wireless network ecosystems and Internet of Things (IoT) connected devices have increased rapidly as technology advances and cyber threats increase. The existing methods cannot make sequential decisions in complex network environments, particularly in scenarios with partial observability and non-stationarity. Network awareness monitors and comprehends the network's assets, vulnerabilities, and ongoing activities in real-time. Advanced analytics, machine learning algorithms, and artificial intelligence are used to improve risk perception by analyzing massive amounts of information, identifying trends, and anticipating future security breaches. Hence, this study suggests the Deep Reinforcement Learning-assisted Network Awareness Risk Perception and Prevention Model (DRL-NARPP) for detecting malicious activity in cybersecurity. The proposed system begins with the concept of network awareness, which uses DRL algorithms to constantly monitor and evaluate the condition of the network in terms of factors like asset configurations, traffic patterns, and vulnerabilities. DRL provides autonomous learning and adaptation to changing network settings, revealing the ever-changing nature of network awareness risks in real time. Incorporating DRL into risk perception increases the system's capacity to recognize advanced attack methods while simultaneously decreasing the number of false positives and enhancing the reliability of risk assessments. DRL algorithms drive dynamic and context-aware response mechanisms, making up the adaptive network prevention component of the development. Predicting new threats and proactively deploying preventive measures, such as changing firewall rules, isolating compromised devices, or dynamically reallocating resources to reduce developing risks, is made possible by the system's ability to learn from historical data and prevailing network activity. The suggested DRL-NARPP model increases the anomaly detection rate by 98.3%, the attack prediction accuracy rate by 97.4%, and the network risk assessment ratio by 96.4%, reducing the false positive ratio by 11.2% compared to other popular methodologies.

A homotopy-based reinforcement learning scheme to optimal control for Markov switched interconnected systems

ABSTRACT In this paper, a homotopy-based reinforcement learning optimal control method is developed for Markov switched interconnected systems with unknown system dynamics. By utilising the subsystem decomposition method and parallel learning control method, the solution of the game coupled algebraic Riccati equations with jumping parameters is approximated. To dispense with the requirement of initial stability, a homotopy-based policy iteration is introduced, which can place unstable poles into a stable plane. In this regard, a model-free reinforcement learning method is presented to design the optimal controller for Markov switched interconnected systems. Finally, the effectiveness of the proposed method is verified by a numerical example and a practical example.

Federated deep reinforcement learning for mobile robot navigation

Navigation of a mobile robot in an unknown environment ensuring the safety of the robot and its surroundings is of utmost importance. Traditional methods, such as pathplanning algorithms, simultaneous localization and mapping, computer vision, and fuzzy techniques, have been employed to address this challenge. However, to achieve better generalization and self-improvement capabilities, reinforcement learning has gained significant attention. The concern of privacy issues in sharing data is also rising in various domains. In this study, a deep reinforcement learning strategy is applied to the mobile robot to move from its initial position to a destination. Specifically, the Deep Q-Learning algorithm has been used for this purpose. This strategy is trained using a federated learning approach to overcome privacy issues and to set a foundation for further analysis of distributed learning. The application scenario considered in this work involves the navigation of a mobile robot to a charging point within a greenhouse environment. The results obtained indicate that both the traditional deep reinforcement learning and federated deep reinforcement learning frameworks are providing 100% success rate. However federated deep reinforcement learning could be a better alternate since it overcomes the privacy issue along with other advantages discussed in this paper.

Physics-constrained robust learning of open-form partial differential equations from limited and noisy data

Unveiling the underlying governing equations of nonlinear dynamic systems remains a significant challenge. Insufficient prior knowledge hinders the determination of an accurate candidate library, while noisy observations lead to imprecise evaluations, which in turn result in redundant function terms or erroneous equations. This study proposes a framework to robustly uncover open-form partial differential equations (PDEs) from limited and noisy data. The framework operates through two alternating update processes: discovering and embedding. The discovering phase employs symbolic representation and a novel reinforcement learning (RL)-guided hybrid PDE generator to efficiently produce diverse open-form PDEs with tree structures. A neural network-based predictive model fits the system response and serves as the reward evaluator for the generated PDEs. PDEs with higher rewards are utilized to iteratively optimize the generator via the RL strategy and the best-performing PDE is selected by a parameter-free stability metric. The embedding phase integrates the initially identified PDE from the discovering process as a physical constraint into the predictive model for robust training. The traversal of PDE trees automates the construction of the computational graph and the embedding process without human intervention. Numerical experiments demonstrate our framework's capability to uncover governing equations from nonlinear dynamic systems with limited and highly noisy data and outperform other physics-informed neural network-based discovery methods. This work opens new potential for exploring real-world systems with limited understanding.

Cooperative traffic optimization with multi-agent reinforcement learning and evolutionary strategy: Bridging the gap between micro and macro traffic control

The emergence of connected and autonomous vehicles (CAVs) holds promise for fine-grained traffic control. However, due to the longevity of future mixed traffic scenarios, there is a need for an in-depth exploration of integrating the microscopic speed control of CAVs with the macroscopic variable speed limit (VSL) of human-driven vehicles (HDVs). This paper proposes a Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME) framework, which combines microscopic CAV control with macroscopic VSL control. The framework incorporates a Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework for intelligent decision-making by microscopic-level vehicles. Additionally, an evolutionary strategy is developed to design the VSL network architecture, enabling macroscopic level real-time speed limit adjustments based on infrastructure. A multi-objective reward function is proposed to optimize both micro and macro efficiency and safety, accounting for both vehicle behavior and traffic flow. Experiments on the designed Bottleneck traffic scenarios show that the proposed approach, CTO-ME, is able to achieve superior performance and outperforms other baselines in terms of traffic throughput, average speed, and safety. Specifically, CTO-ME enhances average velocity by 37%, increases overall throughput by 309%, and raises arrival ratio by 70% than traditional Intelligent Driver Model (IDM).

Reinforcement learning vs rule-based dynamic movement strategies in UAV assisted networks

Since resource allocation of cellular networks is not dynamic, some cells may experience unplanned high traffic demands due to unexpected events. Unmanned aerial vehicles (UAV) can be used to provide the additional bandwidth required for data offloading.Considering real-time and non-real-time traffic classes, our work is dedicated to optimize the placement of UAVs in cellular networks by two approaches. A first rule-based, low complexity method, that can be embedded in the UAV, while the other approach uses Reinforcement Learning (RL). It is based on Markov Decision Processes (MDP) for providing optimal results. The energy of the UAV battery and charging time constraints have been taken into account to cover a typical cellular environment consisting of many cells.We used an open dataset for the Milan cellular network provided by Telecom Italia to evaluate the performance of both proposed models. Considering this dataset, the MDP model outperforms the rule-based algorithm. Nevertheless, the rule-based one requires less processing complexity and can be used immediately without any prior data. This work makes a notable contribution to developing practical and optimal solutions for UAV deployment in modern cellular networks.

Multi-agent reinforcement learning framework based on information fusion biometric ticketing data in different public transport modes

In smart cities, biometric technologies have become extensively used for ticket authentication on public transport. Information fusion plays a key role in biometric ticketing, allowing ticket validation with more data source validation in different public transport modes. This paper proposes a novel biometric technology-based mobile ticket application-based system. We formulate the problem as a multi-agent reinforcement learning framework for biometric ticketing in multi-transport environments. Specifically, we propose the Asynchronous Advantage Critic Biometric Ticketing Framework (A3CBTF) algorithm, which consists of different schemes based on the proposed system. The proposed algorithm framework operates in hybrid transport modes using a parallel reinforcement learning scheme. A key advantage of A3CBTF is that it enables passengers to use a single ticket for various public transport modes. Additionally, even when a passenger’s mobile device is stolen, lost, or has a dead battery, they can still validate their tickets through different information fusion sources, such as fingerprint and face recognition. A3CBTF is a multi-agent system that integrates mobile, transport, edge, and cloud servers to facilitate ticket validation in a distributed environment. By optimizing both convex and concave optimizations, A3CBTF ensures efficient ticket validation with minimal processing time and maximizes validation rewards across different biometric technologies. Experimental results demonstrate that A3CBTF outperforms mobile off with other options such as fingerprint and face recognition in public transport as compared to other ticketing systems.

Integration of blockchain and machine learning for safe and efficient autonomous car systems: A survey

The integration of blockchain and machine learning technologies has the potential to enable the development of more secure, reliable, and efficient autonomous car systems. Blockchain can be used to store, manage, and share the large amounts of data generated by autonomous vehicle various sensors and cameras, ensuring the integrity and security of these data. Machine learning algorithms can be used to analyze and fuse these data in real time, allowing the vehicle to make informed decisions about how to navigate its environment and respond to changing conditions. Thus, the combination of these technologies has the potential to improve the safety, performance, and scalability of autonomous car systems, making them a more applicable and attractive option for consumers and industry stakeholders. In this paper, all relevant technologies, such as machine learning, blockchain and autonomous cars, were explored. Various techniques of machine learning were investigated, including reinforcement learning strategies, the evolution of artificial neural networks and main deep learning algorithms. The main features of the blockchain technology, as well as its different types and consensus mechanisms, were discussed briefly. Autonomous cars, their different types of sensors, potential vulnerabilities, sensor data fusion techniques, and decision-making models were addressed, and main problem domains and trends were underlined. Furthermore, relevant research discussing blockchain for intelligent transportation systems and internet of vehicles was examined. Subsequently, papers related to the integration of blockchain with machine learning for autonomous cars and vehicles were compared and summarized. Finally, the main applications, challenges and future trends of this integration were highlighted.

A novel approach for credit card fraud transaction detection using deep reinforcement learning scheme.

Online transactions are still the backbone of the financial industry worldwide today. Millions of consumers use credit cards for their daily transactions, which has led to an exponential rise in credit card fraud. Over time, many variations and schemes of fraudulent transactions have been reported. Nevertheless, it remains a difficult task to detect credit card fraud in real-time. It can be assumed that each person has a unique transaction pattern that may change over time. The work in this article aims to (1) understand how deep reinforcement learning can play an important role in detecting credit card fraud with changing human patterns, and (2) develop a solution architecture for real-time fraud detection. Our proposed model utilizes the Deep Q network for real-time detection. The Kaggle dataset available online was used to train and test the model. As a result, a validation performance of 97.10% was achieved with the proposed deep learning component. In addition, the reinforcement learning component has a learning rate of 80%. The proposed model was able to learn patterns autonomously based on previous events. It adapts to the pattern changes over time and can take them into account without further manual training.

A Deep Reinforcement Learning Strategy for MEC Enabled Virtual Reality in Telecommunication Networks

A Deep Reinforcement Learning Strategy for MEC Enabled Virtual Reality in Telecommunication Networks

Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration.

When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action-reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.

A zero-shot reinforcement learning strategy for autonomous guidewire navigation

A zero-shot reinforcement learning strategy for autonomous guidewire navigation

Differential graphical game‐based multi‐agent tracking control using integral reinforcement learning

AbstractThis paper studies the cooperative tracking control problem of interacted multi‐agent systems (MASs) under undirected communication. Based on differential graphical game theory, the MAS tracking control problem is formulated as an infinite horizon cooperative differential graphical game‐theoretic tracking control framework, where a multi‐objective optimization problem is designed and then cast into a Pareto‐equivalent single‐objective optimization problem using a scalarization method. Necessary and sufficient conditions for the existence of the Pareto‐optimal strategy to the game theoretic tracking control are established, where it has been proven that the solution to the integral Bellman optimality equation leads to Pareto‐optimal strategy. Then, an off‐policy integral reinforcement learning scheme to find optimal control strategy using a pure data‐driven manner is developed, which consumes less computation efforts than the traditional learning scheme. Simulated results are conducted to validate the effectiveness of the proposed game and IRL‐based tracking control method.

Enhancing Urban Intersection Efficiency: Utilizing Visible Light Communication and Learning-Driven Control for Improved Traffic Signal Performance

This paper introduces an approach to enhance the efficiency of urban intersections by integrating Visible Light Communication (VLC) into a multi-intersection traffic control system. The main objectives include the reduction in waiting times for vehicles and pedestrians, the improvement of overall traffic safety, and the accommodation of diverse traffic movements during multiple signal phases. The proposed system utilizes VLC to facilitate communication among interconnected vehicles and infrastructure. This is achieved by utilizing streetlights, headlamps, and traffic signals for transmitting information. By integrating VLC localization services with learning-driven traffic signal control, the multi-intersection traffic management system is established. A reinforcement learning scheme, based on VLC queuing/request/response behaviors, is utilized to schedule traffic signals effectively. Agents placed at each intersection control traffic lights by incorporating information from VLC-ready cars, including their positions, destinations, and intended routes. The agents devise optimal strategies to improve traffic flow and engage in communication to optimize the collective traffic performance. An assessment of the multi-intersection scenario through the SUMO urban mobility simulator reveals considerable benefits. The system successfully reduces both waiting and travel times. The reinforcement learning approach effectively schedules traffic signals, and the results highlight the decentralized and scalable nature of the proposed method, especially in multi-intersection scenarios. The discussion emphasizes the possibility of applying reinforcement learning in everyday traffic scenarios, showcasing the potential for the dynamic identification of control actions and improved traffic management.

Multi-factor stock trading strategy based on DQN with multi-BiGRU and multi-head ProbSparse self-attention

Reinforcement learning is widely used in financial markets to assist investors in developing trading strategies. However, most existing models primarily focus on simple volume-price factors, and there is a need for further improvement in the returns of stock trading. To address these challenges, a multi-factor stock trading strategy based on Deep Q-Network (DQN) with Multi-layer Bidirectional Gated Recurrent Unit (Multi-BiGRU) and multi-head ProbSparse self-attention is proposed. Our strategy comprehensively characterizes the determinants of stock prices by considering various factors such as financial quality, valuation, and sentiment factors. We first use Light Gradient Boosting Machine (LightGBM) to classify turning points for stock data. Then, in the reinforcement learning strategy, Multi-BiGRU, which holds the bidirectional learning of historical data, is integrated into DQN, aiming to enhance the model’s ability to understand the dynamics of the stock market. Moreover, the multi-head ProbSparse self-attention mechanism effectively captures interactions between different factors, providing the model with deeper market insights. We validate our strategy’s effectiveness through extensive experimental research on stocks from Chinese and US markets. The results show that our method outperforms both temporal and non-temporal models in terms of stock trading returns. Ablation studies confirm the critical role of LightGBM and multi-head ProbSparse self-attention mechanism. The experiment section also demonstrates the significant advantages of our model through the presentation of box plots and statistical tests. Overall, by fully considering the multi-factor data and the model’s feature extraction capabilities, our work is expected to provide investors with more precise trading decision support.Graphical abstract

Learning the sound inventory of a complex vocal skill via an intrinsic reward.

Reinforcement learning (RL) is thought to underlie the acquisition of vocal skills like birdsong and speech, where sounding like one's "tutor" is rewarding. However, what RL strategy generates the rich sound inventories for song or speech? We find that the standard actor-critic model of birdsong learning fails to explain juvenile zebra finches' efficient learning of multiple syllables. However, when we replace a single actor with multiple independent actors that jointly maximize a common intrinsic reward, then birds' empirical learning trajectories are accurately reproduced. The influence of each actor (syllable) on the magnitude of global reward is competitively determined by its acoustic similarity to target syllables. This leads to each actor matching the target it is closest to and, occasionally, to the competitive exclusion of an actor from the learning process (i.e., the learned song). We propose that a competitive-cooperative multi-actor RL (MARL) algorithm is key for the efficient learning of the action inventory of a complex skill.

Mutual information oriented deep skill chaining for multi‐agent reinforcement learning

AbstractMulti‐agent reinforcement learning relies on reward signals to guide the policy networks of individual agents. However, in high‐dimensional continuous spaces, the non‐stationary environment can provide outdated experiences that hinder convergence, resulting in ineffective training performance for multi‐agent systems. To tackle this issue, a novel reinforcement learning scheme, Mutual Information Oriented Deep Skill Chaining (MioDSC), is proposed that generates an optimised cooperative policy by incorporating intrinsic rewards based on mutual information to improve exploration efficiency. These rewards encourage agents to diversify their learning process by engaging in actions that increase the mutual information between their actions and the environment state. In addition, MioDSC can generate cooperative policies using the options framework, allowing agents to learn and reuse complex action sequences and accelerating the convergence speed of multi‐agent learning. MioDSC was evaluated in the multi‐agent particle environment and the StarCraft multi‐agent challenge at varying difficulty levels. The experimental results demonstrate that MioDSC outperforms state‐of‐the‐art methods and is robust across various multi‐agent system tasks with high stability.

RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting

Large Language Models (LLMs) have demonstrated impressive capabilities in creative tasks such as storytelling and E-mail generation. However, as LLMs are primarily trained on final text results rather than intermediate revisions, it might be challenging for them to perform text rewriting tasks. Most studies in the rewriting tasks focus on a particular transformation type within the boundaries of single sentences. In this work, we develop new strategies for instruction tuning and reinforcement learning to better align LLMs for cross-sentence rewriting tasks using diverse wording and structures expressed through natural languages including 1) generating rewriting instruction data from Wiki edits and public corpus through instruction generation and chain-of-thought prompting; 2) collecting comparison data for reward model training through a new ranking function. To facilitate this research, we introduce OpenRewriteEval, a novel benchmark covers a wide variety of rewriting types expressed through natural language instructions. Our results show significant improvements over a variety of baselines.