Control of a Quadrotor With Reinforcement Learning

Reinforcement learning (RL) is learning from interactions with the environment in order to accomplish certain long-term objectives connected to the environmental condition. Reinforcement learning takes place when action sequences, observations, and rewards are used as inputs, and is hypothesis-based and goal-oriented. The purpose of the research was to conduct a systematic literature review of reinforcement algorithms in machine learning in order to develop a successful multi-agent RL algorithm that can be applied to robotics, network packet routing, energy distribution, and other applications. The robotics-related RL techniques of value-based RL, policy-based RL, model-based RL, deep reinforcement learning, meta-RL, and inverse RL were examined. As a result, the robotics-related RL techniques of value-based RL, policy-based RL, model-based RL, deep RL, meta-RL, and inverse RL were discussed in this research work. The asynchronous advantage actor-critic algorithm (A3C) is one of the best reinforcement algorithms. A3C performs better on deep RLchallenges and is quicker and easier to use.

This paper presents a comprehensive exploration of various methodologies and techniques aimed at enhancing tool path planning in CNC machining. It discusses differential vector optimization for generating smooth trajectories, kinematic constraint adjustment to optimize cycle time and minimize cornering errors, and equidistant tool path planning for curved freeform surfaces. Additionally, the paper delves into the integration of reinforcement learning (RL) algorithms, such as dynamic search strategies and deep RL models, to optimize tool path planning. Results showcase significant improvements in convergence rates, learning efficiency, and navigation performance with RL algorithms. Moreover, the synergy between RL and traditional optimization methods, like Artificial Potential Field theory, is highlighted, showing promise in addressing challenges in static workspaces. The paper also discusses the evolution of deep RL techniques over time, suggesting continual advancements in optimizing tool path planning. Overall, the findings underscore the critical role of advanced planning algorithms and RL techniques in enhancing CNC machining processes, paving the way for further advancements in manufacturing efficiency and accuracy.

The rapid evolution of data-intensive applications has intensified the need for efficient and adaptive database systems. Traditional database tuning methods, relying on manual interventions and rule-based optimizations, often fall short in handling dynamic workloads and complex parameter interdependencies. This paper introduces a novel approach to self-tuning database systems using reinforcement learning (RL) techniques, enabling databases to autonomously optimize configurations such as indexing strategies, memory allocation, and query execution plans. The proposed framework significantly enhances performance, scalability, and resource utilization by leveraging RL’s ability to learn from interactions and adapt to changing environments. Experimental evaluations demonstrate up to a 45% improvement in query execution times and superior adaptability to workload variations compared to traditional methods. This study highlights RL's potential to transform database management, setting the stage for next-generation intelligent and autonomous data systems. Modern database systems face increasing complexity due to the diverse workloads and dynamic environments they operate in. Traditional database tuning methods often require significant manual intervention and expertise, making them inefficient for large-scale systems. This paper presents a novel approach to self-tuning database systems using reinforcement learning (RL) techniques. By leveraging RL, databases can autonomously learn and adapt to changing conditions, optimizing configurations such as indexing, query execution plans, and memory allocation. We outline a framework for implementing RL-based self-tuning, discuss key challenges, and evaluate the approach against traditional methods. Results indicate significant improvements in performance, adaptability, and resource utilization, demonstrating the potential of RL for next-generation database systems.

Swarm robotics is the study of how large number of relatively simple physically embodied agents can be designed such that a desired collective behavior emerges from the local interactions among agents and between the agents and the environment. It is a novel approach to the coordination of large numbers of robots. It is inspired from the observation of social insects such as ants, termites, wasps and bees which stand as fascinating examples of how a large number of simple individuals can interact to create collectively intelligent systems. Social insects are known to coordinate their actions to accomplish tasks that are beyond the capabilities of a single individual: termites build large and complex mounds, army ants organize impressive foraging raids and ants can collectively carry large preys. Such coordination capabilities are still beyond the reach of current multirobot systems. In this research, most recent developments of swarm robot systems are addressed by classifying the primary research axes in terms of principal topic areas that have generated significant levels of research. Specific research scopes in each of primary research axes are also classified and the key open issues in this research scopes are identified. With the goal of bringing some objective grounding to the important areas of the mentioned research scopes, this dissertation presents an empirical analysis on multi-agent foraging task using reinforcement learning algorithms. Foraging task is one of the mostly used test applications in multi-agent systems. The aim of agents in foraging task is to find the pucks (prey) and bring them to the home (nest) location in a environment while satisfying the given constraints. This task is also known as prey retrieval task. Reinforcement learning is used to tackle the modeled multi-agent foraging task. Reinforcement learning has been extensively used in many applications such as industrial control, time sequence prediction, robot soccer competition and more. In this thesis, a multi-agent foraging task is modeled using Webots (1996) simulation software and the reinforcement learning algorithms and policies are tested. One of the challenges that arise in reinforcement learning is the exploration and exploitation dilemma. A novel learning policy (FIFO-list learning policy) is proposed and compared against the available learning policies reported in the literature to tackle the exploration and exploitation dilemma. An improved reinforcement learning algorithm (Cautious-Q learning algorithm) is also proposed and the performance is compared with the available learning algorithms. The proposed learning algorithm is a combined strategy of on-policy learning and off-policy learning. The improved learning algorithm and the learning policy is implemented on real environment with Khepera 2 mobile robots and results are presented.

In this paper, a method that utilizes the reinforcement learning (RL) technique is proposed to establish an optimal operation plan to obtain maximum power output from a trigen generator. Trigen is a type of combined heat and power system (CHP) that provides chilling, heating, and power generation, and the turbo expander generator (TEG) is a generator that uses the decompression energy of gas to generate electricity. If the two are combined to form a power source, a power generation system with higher efficiency can be created. However, it is very difficult to control the heat and power generation amount of TEG and trigen according to the flow rate of natural gas that changes every moment. Accordingly, a method is proposed to utilize the RL technique to determine the operation process to attain an even higher efficiency. When the TEG and trigen are configured using the RL technique, the power output can be maximized, and the power output variability can be reduced to obtain high-quality power. When using the RL technique, it was confirmed that the overall efficiency was improved by an additional 3%.

This paper addresses one of the main challenges on the way to an offshore transnational multi-terminal dc (MTdc) network: its control and operation. The main objective is to demonstrate the feasibility of using reinforcement learning (RL) techniques to control, in real time, a multi-terminal dc network aimed at integrating offshore wind farms (OWFs). This method of controlling MTdc networks using RL techniques is called Online Distributed Voltage Control (ODVC). The ODVC strategy uses Continuous Action Reinforcement Learning Automata (CARLA) to optimize power flows in real time. To validate the effectiveness of the proposed control method, dynamic simulations are carried out using a MTdc grid model composed of six nodes, interconnecting three offshore wind farms to three European countries. The results obtained demonstrate the advantages of implementing an online distributed voltage control strategy to obtain feasible controlled power flows with low transmission losses. The results obtained demonstrate the feasibility of the proposed method to control, in real time, MTdc networks and that the RL techniques are well-suited for this problem due to their inherent advantages of coping with stochastic environments.

Reinforcement learning techniques are increasingly utilized in cyber physical systems and traditional control systems, since they allow the controlling logic to learn through its interactions with its environment. However, reinforcement learning techniques have been found to be vulnerable to malicious influence, in the form of so-called adversarial examples, that can lead to, for example, destabilization of the system. In this paper, an optimization method is proposed to provide a directed attack towards a system resulting in destabilization. The attack differs from previous adversarial work against machine learning algorithms in that it focused on cyber physical systems and, in contrast to false-data injection or actuator attacks, assumed that an adversary is able to directly influence the state(s) of the system, to some degree. Furthermore, it is assumed that the system is controlled using a pre-learned optimal policy; i.e., the attack does not poison the learning process but rather leverages imperfections in the learned policy. This means the reinforcement learning algorithm can be vulnerable even while operating under an optimal policy. The optimization approach increases the feasibility of the attack by reducing the overall cost expended by the adversary. This paper describes the theory supporting the attack by proposing an algorithm and its corresponding proof. The attack is validated using OpenAI's gym and the physics simulator Mujoco to simulate the attack on a cyber physical system trained using a deep reinforcement learning method.

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of reinforcement learning in various sectors, such as healthcare and cyber-security, among others. However, reinforcement learning can be time-consuming because the learning algorithms have to determine the long term consequences of their actions using delayed feedback or rewards. Reward shaping is a method of incorporating domain knowledge into reinforcement learning so that the algorithms are guided faster towards more promising solutions. Under an overarching theme of episodic reinforcement learning, this paper shows a unifying analysis of potential-based reward shaping which leads to new theoretical insights into reward shaping in both model-free and model-based algorithms, as well as in multi-agent reinforcement learning.

Cognitive automation represents the next frontier in Robotic Process Automation (RPA), enabling systems to learn, adapt, and optimize decision-making processes dynamically. Traditional RPA platforms, such as UiPath and Automation Anywhere, excel in automating rule-based tasks but lack the ability to handle complex, evolving scenarios that require adaptive intelligence. Integrating reinforcement learning (RL) techniques into RPA workflows offers a transformative approach to enhancing cognitive automation capabilities. RL enables bots to make intelligent, data-driven decisions by learning from their environment, optimizing workflows, and improving operational efficiency over time. This study explores the integration of RL algorithms within UiPath and Automation Anywhere to develop self-learning automation systems capable of handling non-deterministic processes. Key applications include intelligent exception handling, dynamic process optimization, and adaptive customer service automation. By leveraging RL-based decision models, RPA bots can continuously improve their performance, reduce error rates, and optimize workflows beyond predefined rules. The research also examines challenges such as computational complexity, model interpretability, and integration barriers within enterprise automation environments. Solutions such as cloud-based reinforcement learning frameworks, hybrid AI-RPA architectures, and explainable AI techniques are proposed to mitigate these challenges. The findings indicate that reinforcement learning can significantly enhance cognitive automation in RPA, enabling businesses to achieve higher levels of efficiency, adaptability, and intelligent decision-making.

This Reinforcement learning (RL) techniques are increasingly employed for optimal tool path planning in CNC machining to improve efficiency and reduce costs. Traditional RL algorithms, like Policy Iteration, encounter challenges due to the vast design parameter search space. This has led to the development of innovative approaches such as auto-tuner-based ordinal regression methods, offering accelerated parameter exploration and faster convergence towards optimal policies. Lin et al. present a systematic solution for complex cavity milling tool path generation using RL, demonstrating superior speed and quality compared to evolutionary computational techniques. The shift towards RL-based approaches signifies a paradigm change in tool path optimization, emphasizing its potential to enhance machining efficiency and accuracy. This paper compares various RL algorithms and approaches, showcasing their effectiveness in optimizing tool paths. Results indicate improvements in machining efficiency, accuracy, and operational costs through the integration of RL models into CNC systems. Overall, RL-based optimization offers competitive advantages, aiding stakeholders in making informed decisions for efficient CNC machining operations.

Recent advances in machine learning have demonstrated the effectiveness of Conditional Generative Adversarial Network (CGAN) and Reinforcement Learning (RL) techniques in a variety of domains, from image synthesis to decision-making tasks. However, their integration and application to the generation of contextual datasets remains under explored. This paper proposes a novel approach to combine CGAN and RL to increase dataset accuracy in specific context domains. The proposed methodology focuses on using CGANs to generate synthetic data that closely mimic real-world contextual variations corresponding to Advanced Persistent Threat (APT) attacks. By adapting the generator to contextual variables such as environmental conditions or attacker behavior, the generated data are more closely aligned to the target distribution, improving model robustness and generalization. In addition, RL techniques are used to iteratively refine the generated data samples, leading the generator to produce samples that not only adhere to the desired context, but also match the specific objectives of the proposed work. The main contributions of this work include the development of a unified framework that seamlessly integrates CGAN and RL for generating contextual datasets to strengthen Deep Learning Models. The proposed algorithms have been implemented using python programming language along with APIs. Experimental results demonstrate significant improvements in accuracy and reliability compared to traditional dataset augmentation methods.  

This article dedicates to investigating a methodology for enhancing adaptability to environmental changes of reinforcement learning (RL) techniques with data efficiency, by which a joint control protocol is learned using only data for multiagent systems (MASs). Thus, all followers are able to synchronize themselves with the leader and minimize their individual performance. To this end, an optimal synchronization problem of heterogeneous MASs is first formulated, and then an arbitration RL mechanism is developed for well addressing key challenges faced by the current RL techniques, that is, insufficient data and environmental changes. In the developed mechanism, an improved Q-function with an arbitration factor is designed for accommodating the fact that control protocols tend to be made by historic experiences and instinctive decision-making, such that the degree of control over agents' behaviors can be adaptively allocated by on-policy and off-policy RL techniques for the optimal multiagent synchronization problem. Finally, an arbitration RL algorithm with critic-only neural networks is proposed, and theoretical analysis and proofs of synchronization and performance optimality are provided. Simulation results verify the effectiveness of the proposed method.

Modern power systems are characterized by bidirectional power flows, smart metering schemes capable of net metering applications and several renewable energy-based distributed generators whose outputs are stochastic. Moreover, customers are producing their own energy using renewables and are involved in contractual energy cooperation with the main grid. This introduces more complexities in power management as the number of random variables to be considered in scheduling increases. Therefore, there is a need to develop more robust power management algorithms that are adaptive to the stochasticity in the new power systems. Recently, the employment of reinforcement learning (RL) techniques in power management and scheduling has received significant interest. This paper reviews recent RL methods used in solving grid-tied microgrid power control problems and highlights the potential benefits and disadvantages of RL algorithms in solving these problems. The various challenges associated with the control of power flow in microgrids being linked to the utility grids are first described using electric vehicle charging station that is powered by PV and the utility grid as an example. The application of various RL techniques in addressing those challenges is then critically reviewed.

Comprehensive review of reinforcement learning in lung cancer diagnosis and treatment: Taxonomy, challenges and recommendations

In this study, a systematic review on production scheduling based on reinforcement learning (RL) techniques using especially bibliometric analysis has been carried out. The aim of this work is, among other things, to point out the growing interest in this domain and to outline the influence of RL as a type of machine learning on production scheduling. To achieve this, the paper explores production scheduling using RL by investigating the descriptive metadata of pertinent publications contained in Scopus, ScienceDirect, and Google Scholar databases. The study focuses on a wide spectrum of publications spanning the years between 1996 and 2024. The findings of this study can serve as new insights for future research endeavors in the realm of production scheduling using RL techniques.