Reinforcement Learning Architecture Research Articles

An investigation of belief-free DRL and MCTS for inspection and maintenance planning

We propose a novel Deep Reinforcement Learning (DRL) architecture for sequential decision processes under uncertainty, as encountered in inspection and maintenance (I &M) planning. Unlike other DRL algorithms for (I &M) planning, the proposed +RQN architecture dispenses with computing the belief state and directly handles erroneous observations instead. We apply the algorithm to a basic I &M planning problem for a one-component system subject to deterioration. In addition, we investigate the performance of Monte Carlo tree search for the I &M problem and compare it to the +RQN. The comparison includes a statistical analysis of the two methods’ resulting policies, as well as their visualization in the belief space.

Transferable multi-objective factory layout planning using simulation-based deep reinforcement learning

Factory layout planning aims at finding an optimized layout configuration under consideration of varying influences such as the material flow characteristics. Manual layout planning can be characterized as a complex decision-making process due to a large number of possible placement options. Automated planning approaches aim at reducing the manual planning effort by generating optimized layout variants in the early stages of layout planning. Recent developments have introduced deep Reinforcement Learning (RL) based planning approaches that allow to optimize a layout under consideration of a single optimization criterion. However, within layout planning, multiple partially conflicting planning objectives have to be considered. Such multiple objectives are not considered by existing RL-based approaches. This paper addresses this research gap by presenting a novel deep RL-based layout planning approach that allows consideration of multiple objectives for optimization. Furthermore, existing RL-based planning approaches only consider analytically formulated objectives such as the transportation distance. Consequently, dynamic influences in the material flow are neglected which can result in higher operational costs of the future factory. To address this issue, a discrete event simulation module is developed that allows simulating manufacturing and material flow processes simultaneously for any layout configuration generated by the RL approach. Consequently, the presented approach considers material flow simulation results for multi-objective optimization. To investigate the capabilities of RL-based factory layout planning, different RL architectures are compared based on a simplified application scenario. Throughput time, media supply, and material flow clarity are considered as optimization objectives. The best performing architecture is then applied to an exemplary application scenario and compared with the results obtained by a combined version of the genetic algorithm and tabu search, the non-dominated sorting genetic algorithm, and the optimal solution. Finally, two industrial planning scenarios, one focusing on brownfield and one on greenfield planning, are considered. The results show that the performance of RL compared to meta-heuristics depends on the considered computation time. With time the results generated by the RL approach exceed the quality of the best conventional solution by up to 11%. Finally, the potential of applying transfer learning is investigated for three different application scenarios. It is observed that RL can learn generalized patterns for factory layout planning, which allows to significantly reduce the required training time and can lead to improved solution quality. Thus, the use of pre-trained RL models shows a substantial performance potential for automated factory layout planning which cannot be achieved with conventional automated planning approaches.

Time-Varying Weights in Multi-Reward Architecture for Deep Reinforcement Learning

Time-Varying Weights in Multi-Reward Architecture for Deep Reinforcement Learning

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

Extensive research has been carried out on reinforcement learning methods. The core idea of reinforcement learning is to learn methods by means of trial and error, and it has been successfully applied to robotics, autonomous driving, gaming, healthcare, resource management, and other fields. However, when building reinforcement learning solutions at the edge, not only are there the challenges of data-hungry and insufficient computational resources but also there is the difficulty of a single reinforcement learning method to meet the requirements of the model in terms of efficiency, generalization, robustness, and so on. These solutions rely on expert knowledge for the design of edge-side integrated reinforcement learning methods, and they lack high-level system architecture design to support their wider generalization and application. Therefore, in this paper, instead of surveying reinforcement learning systems, we survey the most commonly used options for each part of the architecture from the point of view of integrated application. We present the characteristics of traditional reinforcement learning in several aspects and design a corresponding integration framework based on them. In this process, we show a complete primer on the design of reinforcement learning architectures while also demonstrating the flexibility of the various parts of the architecture to be adapted to the characteristics of different edge tasks. Overall, reinforcement learning has become an important tool in intelligent decision making, but it still faces many challenges in the practical application in edge computing. The aim of this paper is to provide researchers and practitioners with a new, integrated perspective to better understand and apply reinforcement learning in edge decision-making tasks.

Introduction to Machine Learning

Machine learning has emerged as a transformative technology with wide-ranging applications across diverse domains, from healthcare and finance to autonomous systems and environmental monitoring. This paper provides a comprehensive review of the field of machine learning, encompassing foundational concepts, recent advancements, applications, ethical considerations, challenges, and future directions. The paper begins with an overview of machine learning, highlighting its historical evolution, importance, and applications in various sectors. It then delves into the foundational principles of machine learning, including different types of learning, model evaluation techniques, and statistical foundations. Advances in machine learning algorithms, such as supervised learning, unsupervised learning, reinforcement learning, and deep learning architectures, are discussed in detail. The paper explores the wide-ranging applications of machine learning across different domains, including healthcare, finance, autonomous systems, natural language processing, image and video processing, and environmental monitoring. Ethical and societal implications of machine learning, such as bias and fairness, privacy concerns, automation and job displacement, responsible AI development, and regulatory frameworks, are examined. Challenges and future directions in machine learning, including data quality and quantity issues, interpretability and explainability of models, scalability and resource constraints, addressing ethical and societal concerns, advancements in interdisciplinary research, and opportunities for innovation and collaboration, are discussed. Overall, this paper provides a comprehensive overview of machine learning, highlighting its transformative potential, ethical considerations, challenges, and opportunities for future research and development. It serves as a valuable resource for researchers, practitioners, policymakers, and stakeholders interested in understanding the evolving landscape of machine learning and its impact on society

Deep reinforcement learning architectures for automatic organ segmentation

In the automated segmentation field, the most promising results were obtained using Deep learning (DL) architectures that employ a mix of “Fully convolutional neural networks” (FCN), “Generative adversarial networks” (GAN), or “Recurrent neural networks” (RNN). In our paper we aim to challenge the status-quo of using supervised learning algorithms by proposing and implementing different Deep Reinforcement learning (DRL) architectures that execute the same task – human organ automatic segmentation. We present a completely new approach that defines the segmentation as a Markov Decision Process (MDP) problem where an agent navigates in the environment (CTs) and learns a policy to delineate the contour of the human organ. For solving this problem, we utilized different architectures built on DRL agents. The first architecture employs the Deep Q-Network (DQN) algorithm and uses discrete actions to navigate through CTs while executing the automatic segmentation for the heart. The second architecture implements the Proximal Policy Optimization (PPO) algorithm and utilizes a continuous action space in order to execute the same segmentation process. We tested these two architectures in different setups and using varied reward systems and our results prove that DRL can be used successfully for automated medical segmentation. We also proved that both off-policy, discrete action algorithms or on-policy, continuous action algorithms converge to generalized results.

Learning to Walk With Deep Reinforcement Learning: Forward Dynamic Simulation of a Physics-Based Musculoskeletal Model of an Osseointegrated Transfemoral Amputee.

This paper leverages the OpenSim physics-based simulation environment for the forward dynamic simulation of an osseointegrated transfemoral amputee musculoskeletal model, wearing a generic prosthesis. A deep reinforcement learning architecture, which combines the proximal policy optimization algorithm with imitation learning, is designed to enable the model to walk by using three different observation states. The first is a complete state that includes the agent's kinematics, ground reaction forces, and muscle data; the second is a reduced state that only includes the kinematics and ground reaction forces; the third is an augmented state that combines the kinematics and ground reaction forces with a prediction of the muscle data generated by a fully-connected feed-forward neural network. The empirical results demonstrate that the model trained with the augmented observation state can achieve walking patterns with rewards and gait symmetry ratings comparable to those of the model trained with the complete observation state, while there are no symmetric walking patterns when using the reduced observation state. This paper shows the importance of including muscle data in a deep reinforcement learning architecture for the forward dynamic simulation of musculoskeletal models of transfemoral amputees.

An Efficient Distributed Reinforcement Learning Architecture for Long-haul Communication between Actors and Learner

An Efficient Distributed Reinforcement Learning Architecture for Long-haul Communication between Actors and Learner

Multi-objective reinforcement learning for autonomous drone navigation in urban areas with wind zones

Drones can be used for tasks such as data collection and logistics in civil engineering. Current research on autonomous drones mainly focuses on planning a safe path and avoiding obstacles in a static environment. However, navigating a drone in complex environments like urban areas involves many dynamic constraints, such as building layout, winds, and signal coverage, which are interdependent. The wind factor is the most important among these environmental factors, which may cause a drone to lose control or even crash. This paper presents a multi-objective navigation reinforcement learning algorithm (MONRL) for the drone to navigate and avoid obstacles in an unknown environment when dynamic wind zones present, with only imagery data about the building layouts. Based on a deep reinforcement learning and memory architecture, the drone develops policies to prioritize navigation decisions, optimizing the path while minimizing negative impact of winds with only sparse sensor data, in our case, camera inputs. By leveraging the advantages of the proposed method in estimating the environmental factors from previous trials, no aerodynamic force sensors are needed for the drone to develop effective strategies to navigate to target while counteracting to milder winds and dodging aways from stronger winds. The proposed method was tested in a virtual environment, and a real model of New York City. The method is expected to contribute to new automation algorithms for urban aerial logistics and future automated civil infrastructure inspection.

HiMAQ: Hierarchical multi-agent Q-learning-based throughput and fairness improvement for UAV-Aided IoT networks

Recently, various types of Internet of Things (IoT) services have become wide spread, and new types of IoT devices are emerging. However, the significant number of high-rise buildings in urban environments renders it difficult to provide seamless and high-speed network connectivity consistently to these IoT devices via existing terrestrial IoT networks. Therefore, we herein consider unmanned aerial vehicle (UAV)-aided IoT networks, in which a ground base station (GBS) and a UAV base station (UBS) coexist. In this paper, a hierarchical multi-agent Q-learning (HiMAQ) framework is proposed to maximize the system throughput while minimizing device outage in UAV-aided IoT networks. The proposed HiMAQ adopts distributed multi-agent inner-loop reinforcement learning (RL) for determining the optimal transmission power of UBSs and GBSs, and distributed outer-loop RL to determine the optimal UBS deployment. This hierarchical RL architecture can reduce the computational complexity of the proposed RL approach compared with the centralized RL approach. The performance of HiMAQ in various network conditions is demonstrated considering the mobility of IoT devices and even/uneven spatial traffic distributions. Simulation results show that the proposed HiMAQ can maximize the system throughput and minimize the number of outage devices compared with various benchmark algorithms.

A Novel Approach for Train Tracking in Virtual Coupling Based on Soft Actor-Critic

The development of virtual coupling technology provides solutions to the challenges faced by urban rail transit systems. Train tracking control is a crucial component in the operation of virtual coupling, which plays a pivotal role in ensuring the safe and efficient movement of trains within the train and along the rail network. In order to ensure the high efficiency and safety of train tracking control in virtual coupling, this paper proposes an optimization algorithm based on Soft Actor-Critic for train tracking control in virtual coupling. Firstly, we construct the train tracking model under the reinforcement learning architecture using the operation states of the train, Proportional Integral Derivative (PID) controller output, and train tracking spacing and speed difference as elements of reinforcement learning. The train tracking control reward function is designed. Then, the Soft Actor-Critic (SAC) algorithm is used to train the virtual coupling train tracking reinforcement learning model. Finally, we took the Deep Deterministic Policy Gradient as the comparison algorithm to verify the superiority of the algorithm proposed in this paper.

Cogni-Sec: A secure cognitive enabled distributed reinforcement learning model for medical cyber–physical system

The advent of the Internet of Things (IoT) has resulted in significant technical development in the healthcare sector, enabling the establishment of Medical Cyber–Physical Systems (MCPS). The increased number of MCPS generates a massive amount of privacy-sensitive data, hence it is important to enhance the security of devices and data transmission in MCPS. Earlier several research studies were undertaken in order to enhance security in healthcare, but none of them could adapt to changing behaviors of data attacks. Here the role of blockchain and Reinforcement Learning (RL) comes into play since it can adjust itself to the nature of changing attacks, thus preventing any kind of attacks. This work proposes a solution, named Cogni-Sec, which employs a decentralized cognitive blockchain and Reinforcement Learning architecture and addresses the security issue. Blockchain is incorporated in the approach for data storage and transmission to increase the degree of security in the MCPS modules. Hyperledger Fabric is applied as the blockchain base which shows transaction query results with nearly 10% increased throughput, 69% less memory consumption, and 15% lower CPU usage when compared to Ethereum. Further security risk at the block mining level within a blockchain network is reduced by introducing distributed Reinforcement Learning architecture in replacement for the miner nodes, which imitates the cognitive behavior of miners in a distributed environment. Different multi-agent learning systems have been evaluated for building the mining agent. Among these, the a3c agent in distributed learning setup yields the optimum cumulative reward with a median value of 54.5 and minimizes the maximum number of data threats.

Deep reinforcement learning based planning method in state space for lunar rovers

The unmanned lunar rover is essential for lunar exploration and construction. Executing environment differ from what humans get since communication needs time from Earth to Moon. Considering possible discrepancies between the pre-considered environment from the planner and the real environment for sampling tasks on the moon, a planner that generates short plans quickly should be used. Therefore, a planner for both standard and emergency planning based on deep reinforcement learning (DRL) is demonstrated in this paper. This planner can create a full-range plan 13.5 times faster than the traditional planner on complex problems or 10.1 times faster while controlling the rover step-by-step in the state space. Based on a specific moon sampling scenario, we propose a tracking reward guiding the rover searching in the states in the deep reinforcement learning architecture which is presented and created by a state space representation by matrix, randomly available training state pairs and the plans generated by a custom breadth-first search (BFS) planner for the tracking reward. The BFS planner obtains a custom state hash algorithm and a preparation to train state pairs for safety and flexibility. Tests on training and planning are performed to validate the effectiveness, robustness and customization of the proposed method in a planning domain with multiple rovers. Our model can handle three kinds of emergencies, even if they occur frequently. The success rate is beyond the state-of-the-art model. While facing emergencies, the average response time of our model is 324 times faster than the classical planner.

Research on energy-saving driving control of hydrogen fuel bus based on deep reinforcement learning in freeway ramp weaving area

In the intelligent network traffic scenario, the convenient acquisition of microscopic vehicle states and global traffic states can help solve the problems of vehicle driving and energy management in complex traffic environments. This paper proposes a new energy management method for the hydrogen fuel cell bus based on the double-layer deep deterministic policy gradient (DDPG). Combined with the SUMO simulation platform, a double-layer deep reinforcement learning (D-DRL) architecture based on DDPG is designed to improve control accuracy and training speed. In the upper D-DRL, the Agent handles the effects of complex traffic environments to control the reasonable speed of the vehicle and keep it running smoothly to reduce the loss of energy caused by speed changes, compared with the SUMO-IDM model, the maximum-minimum velocity difference was reduced by 21 % and the acceleration and acceleration change was reduced by 7.9 % and 19 %, After the Agent receives the output speed from the upper layer, it distributes the power between fuel cell and power cell. Compared with the DP algorithm, it keeps the SOC at a higher level, the hydrogen consumption level reaches 93.25 %, and the fluctuation amplitude decreases by 42.09 %, effectively improving fuel cell durability.

Comparing deep reinforcement learning architectures for autonomous racing

In classical autonomous racing, a perception, planning, and control pipeline is employed to navigate vehicles around a track as quickly as possible. In contrast, neural network controllers have been used to replace either part of or the entire pipeline. This paper compares three deep learning architectures for F1Tenth autonomous racing: full planning, which replaces the global and local planner, trajectory tracking, which replaces the local planner and end-to-end, which replaces the entire pipeline. The evaluation contrasts two reward signals, compares the DDPG, TD3 and SAC algorithms and investigates the generality of the learned policies to different test maps. Training the agents in simulation shows that the full planning agent has the most robust training and testing performance. The trajectory tracking agents achieve fast lap times on the training map but low completion rates on different test maps. Transferring the trained agents to a physical F1Tenth car reveals that the trajectory tracking and full planning agents transfer poorly, displaying rapid side-to-side swerving (slaloming). In contrast, the end-to-end agent, the worst performer in simulation, transfers the best to the physical vehicle and can complete the test track with a maximum speed of 5 m/s. These results show that planning methods outperform end-to-end approaches in simulation performance, but end-to-end approaches transfer better to physical robots.

Leveraging Minimum Nodes for Optimum Key Player Identification in Complex Networks: A Deep Reinforcement Learning Strategy with Structured Reward Shaping

The problem of finding key players in a graph, also known as network dismantling, or network disintegration, aims to find an optimal removal sequence of nodes (edges, substructures) through a certain algorithm, ultimately causing functional indicators such as the largest connected component (GCC) or network pair connectivity in the graph to rapidly decline. As a typical NP-hard problem on graphs, recent methods based on reinforcement learning and graph representation learning have effectively solved such problems. However, existing reinforcement-learning-based key-player-identification algorithms often need to remove too many nodes in order to achieve the optimal effect when removing the remaining network until no connected edges remain. The use of a minimum number of nodes while maintaining or surpassing the performance of existing methods is a worthwhile research problem. To this end, a novel algorithm called MiniKey was proposed to tackle such challenges, which employs a specific deep Q-network architecture for reinforcement learning, a novel reward-shaping mechanism based on network functional indicators, and the graph-embedding technique GraphSage to transform network nodes into latent representations. Additionally, a technique dubbed ‘virtual node technology’ is integrated to grasp the overarching feature representation of the whole network. This innovative algorithm can be effectively trained on small-scale simulated graphs while also being scalable to large-scale real-world networks. Importantly, experiments from both six simulated datasets and six real-world datasets demonstrates that MiniKey can achieve optimal performance, striking a perfect balance between the effectiveness of key node identification and the minimization of the number of nodes that is utilized, which holds potential for real-world applications such as curbing misinformation spread in social networks, optimizing traffic in transportation systems, and identifying key targets in biological networks for targeted interventions.

Bridging Locomotion and Manipulation Using Reconfigurable Robotic Limbs via Reinforcement Learning.

Locomotion and manipulation are two essential skills in robotics but are often divided or decoupled into two separate problems. It is widely accepted that the topological duality between multi-legged locomotion and multi-fingered manipulation shares an intrinsic model. However, a lack of research remains to identify the data-driven evidence for further research. This paper explores a unified formulation of the loco-manipulation problem using reinforcement learning (RL) by reconfiguring robotic limbs with an overconstrained design into multi-legged and multi-fingered robots. Such design reconfiguration allows for adopting a co-training architecture for reinforcement learning towards a unified loco-manipulation policy. As a result, we find data-driven evidence to support the transferability between locomotion and manipulation skills using a single RL policy with a multilayer perceptron or graph neural network. We also demonstrate the Sim2Real transfer of the learned loco-manipulation skills in a robotic prototype. This work expands the knowledge frontiers on loco-manipulation transferability with learning-based evidence applied in a novel platform with overconstrained robotic limbs.

EDA3-X: Distributed Attentional Actor Architecture for Interpretability of Coordinated Behaviors in Multi-Agent Systems

In this paper, we propose an enhanced version of the distributed attentional actor architecture (eDA3-X) for model-free reinforcement learning. This architecture is designed to facilitate the interpretability of learned coordinated behaviors in multi-agent systems through the use of a saliency vector that captures partial observations of the environment. Our proposed method, in principle, can be integrated with any deep reinforcement learning method, as indicated by X, and can help us identify the information in input data that individual agents attend to during and after training. We then validated eDA3-X through experiments in the object collection game. We also analyzed the relationship between cooperative behaviors and three types of attention heatmaps (standard, positional, and class attentions), which provided insight into the information that the agents consider crucial when making decisions. In addition, we investigated how attention is developed by an agent through training experiences. Our experiments indicate that our approach offers a promising solution for understanding coordinated behaviors in multi-agent reinforcement learning.

Hierarchical Model-Based Deep Reinforcement Learning for Single-Asset Trading

We present a hierarchical reinforcement learning (RL) architecture that employs various low-level agents to act in the trading environment, i.e., the market. The highest-level agent selects from among a group of specialized agents, and then the selected agent decides when to sell or buy a single asset for a period of time. This period can be variable according to a termination function. We hypothesized that, due to different market regimes, more than one single agent is needed when trying to learn from such heterogeneous data, and instead, multiple agents will perform better, with each one specializing in a subset of the data. We use k-meansclustering to partition the data and train each agent with a different cluster. Partitioning the input data also helps model-based RL (MBRL), where models can be heterogeneous. We also add two simple decision-making models to the set of low-level agents, diversifying the pool of available agents, and thus increasing overall behavioral flexibility. We perform multiple experiments showing the strengths of a hierarchical approach and test various prediction models at both levels. We also use a risk-based reward at the high level, which transforms the overall problem into a risk-return optimization. This type of reward shows a significant reduction in risk while minimally reducing profits. Overall, the hierarchical approach shows significant promise, especially when the pool of low-level agents is highly diverse. The usefulness of such a system is clear, especially for human-devised strategies, which could be incorporated in a sound manner into larger, powerful automatic systems.

Nonfragile Output Feedback Tracking Control for Markov Jump Fuzzy Systems Based on Integral Reinforcement Learning Scheme.

In this article, a novel integral reinforcement learning (RL)-based nonfragile output feedback tracking control algorithm is proposed for uncertain Markov jump nonlinear systems presented by the Takagi-Sugeno fuzzy model. The problem of nonfragile control is converted into solving the zero-sum games, where the control input and uncertain disturbance input can be regarded as two rival players. Based on the RL architecture, an offline parallel output feedback tracking learning algorithm is first designed to solve fuzzy stochastic coupled algebraic Riccati equations for Markov jump fuzzy systems. Furthermore, to overcome the requirement of a precise system information and transition probability, an online parallel integral RL-based algorithm is designed. Besides, the tracking object is achieved and the stochastically asymptotic stability, and expected H∞ performance for considered systems is ensured via the Lyapunov stability theory and stochastic analysis method. Furthermore, the effectiveness of the proposed control algorithm is verified by a robot arm system.