Parallel Reinforcement Learning Research Articles

Multi-agent reinforcement learning framework based on information fusion biometric ticketing data in different public transport modes

In smart cities, biometric technologies have become extensively used for ticket authentication on public transport. Information fusion plays a key role in biometric ticketing, allowing ticket validation with more data source validation in different public transport modes. This paper proposes a novel biometric technology-based mobile ticket application-based system. We formulate the problem as a multi-agent reinforcement learning framework for biometric ticketing in multi-transport environments. Specifically, we propose the Asynchronous Advantage Critic Biometric Ticketing Framework (A3CBTF) algorithm, which consists of different schemes based on the proposed system. The proposed algorithm framework operates in hybrid transport modes using a parallel reinforcement learning scheme. A key advantage of A3CBTF is that it enables passengers to use a single ticket for various public transport modes. Additionally, even when a passenger’s mobile device is stolen, lost, or has a dead battery, they can still validate their tickets through different information fusion sources, such as fingerprint and face recognition. A3CBTF is a multi-agent system that integrates mobile, transport, edge, and cloud servers to facilitate ticket validation in a distributed environment. By optimizing both convex and concave optimizations, A3CBTF ensures efficient ticket validation with minimal processing time and maximizes validation rewards across different biometric technologies. Experimental results demonstrate that A3CBTF outperforms mobile off with other options such as fingerprint and face recognition in public transport as compared to other ticketing systems.

Reinforcement learning‐based composite suboptimal control for Markov jump singularly perturbed systems with unknown dynamics

In this article, a model‐free parallel reinforcement learning method is proposed to solve the suboptimal control problem for the Markov jump singularly perturbed systems. First, since fast and slow dynamics coexist in Markov jump singularly perturbed systems, it may lead to ill‐conditioned numerical problems during the controller design process. Therefore, the original system can be decomposed into independent subsystems at different time‐scales by employing the reduced order method. In addition, two model‐free parallel reinforcement learning algorithms are designed to obtain the optimal controllers for the fast and slow subsystems, respectively. Moreover, within the framework of reinforcement learning, the controllers of the Markov jump singularly perturbed systems can be acquired without the systems dynamics. Finally, a numerical example is introduced to prove the effectiveness of proposed algorithms.

Parallel-Reinforcement-Learning-Based Online Energy Management Strategy for Energy Storage Traction Substations in Electrified Railroad

The traction power supply system is the only source of power for electric locomotives. The huge power fluctuations and complex operating conditions of the traction power supply system pose a challenge to the efficient operation of energy storage traction substations. The existing energy management strategies are difficult to achieve accurate charging and discharging, difficult to modify the control rules in real-time, and have poor migration capability. For comparison, the reinforcement learning algorithms can address the shortcomings of rule-based energy management strategies due to their model-free feature. Therefore, this paper proposes an energy management strategy based on parallel reinforcement learning to improve the efficiency of energy utilization while speeding up the convergence of the algorithm. More specifically, a Markov decision framework is established for capturing the energy management process. The Monte Carlo sampling process is also improved to achieve offline optimization by parallel reinforcement learning algorithms and reduce the impact of low-value power fragments on iteration speed. Meanwhile, the algorithm is modified to enable online updates. The case study shows that compared with other energy management strategies, the parallel reinforcement learning-based energy management strategy has faster convergence speed, higher energy exchange efficiency, better migration capability, and can adapt to various complex working conditions.

PDPoSt+sPBFT: A High Performance Blockchain-Assisted Parallel Reinforcement Learning in Industrial Edge-Cloud Collaborative Network

With the increasing demand for resource scheduling efficiency in Industrial Internet of Things (IIoT), parallel reinforcement learning (PRL) based distributed edge-cloud collaborative resource scheduling scheme has attracted enormous attention. However, the computing and communication capacities, the security degree of massive distributed edge computing servers are different. It is difficult to make a large number of edge servers carry out security and efficiency PRL based edge-cloud collaboration resource scheduling scheme. Thus, in this paper, a large-scale distributed edge-cloud collaborative resource scheduling method based on picture delegated proof of state and suspicious practical byzantine fault tolerance (pDPoSt+sPBFT) consensus algorithm is proposed. To be specific, we first propose a collaborative edge-cloud industrial network architecture to support massive industrial intelligence tasks, then a distributed PRL based resource allocation scheme is utilized. Secondly, in order to improve the efficiency and security of distributed PRL training, we propose a server filtering strategy based on pDPoSt algorithm. Finally, a sPBFT algorithm is proposed to further realize security parameter aggregation of distributed PRL. Experimental results show that the proposed method has good efficiency and security performance compared with the traditional distributed edge-cloud collaborative resource scheduling algorithm. The proposed approach has great potential in complex IIoT scenarios.

Curriculum-Based Asymmetric Multi-Task Reinforcement Learning.

We introduce CAMRL, the first curriculum-based asymmetric multi-task learning (AMTL) algorithm for dealing with multiple reinforcement learning (RL) tasks altogether. To mitigate the negative influence of customizing the one-off training order in curriculum-based AMTL, CAMRL switches its training mode between parallel single-task RL and asymmetric multi-task RL (MTRL), according to an indicator regarding the training time, the overall performance, and the performance gap among tasks. To leverage the multi-sourced prior knowledge flexibly and to reduce negative transfer in AMTL, we customize a composite loss with multiple differentiable ranking functions and optimize the loss through alternating optimization and the Frank-Wolfe algorithm. The uncertainty-based automatic adjustment of hyper-parameters is also applied to eliminate the need of laborious hyper-parameter analysis during optimization. By optimizing the composite loss, CAMRL predicts the next training task and continuously revisits the transfer matrix and network weights. We have conducted experiments on a wide range of benchmarks in multi-task RL, covering Gym-minigrid, Meta-world, Atari video games, vision-based PyBullet tasks, and RLBench, to show the improvements of CAMRL over the corresponding single-task RL algorithm and state-of-the-art MTRL algorithms. The code is available at: https://github.com/huanghanchi/CAMRL.

Bandit Approach to Conflict-Free Parallel Q-Learning in View of Photonic Implementation

Recently, extensive studies on photonic reinforcement learning to accelerate the process of calculation by exploiting the physical nature of light have been conducted. Previous studies utilized quantum interference of photons to achieve collective decision-making without choice conflicts when solving the competitive multi-armed bandit problem, a fundamental example in reinforcement learning. However, the bandit problem deals with a static environment where the agent’s actions do not influence the reward probabilities. This study aims to extend the conventional approach to a more general type of parallel reinforcement learning targeting the grid world problem. Unlike the conventional approach, the proposed scheme deals with a dynamic environment where the reward changes because of the agent’s actions. A successful photonic reinforcement learning scheme requires both a photonic system that contributes to the quality of learning and a suitable algorithm. This study proposes a novel learning algorithm, a modified bandit Q-learning method, in view of a potential photonic implementation. Here, state–action pairs in the environment are regarded as slot machines in the context of the bandit problem and a change in Q -value is regarded as the reward of the bandit problem. We perform numerical simulations to validate the effectiveness of the bandit algorithm. In addition, we propose a parallel architecture in which multiple agents are indirectly connected through quantum interference of light and quantum principles ensure the conflict-free property of state–action pair selections among agents. We demonstrate that parallel reinforcement learning can be accelerated owing to conflict avoidance among multiple agents.

Collaborative clustering parallel reinforcement learning for edge-cloud digital twins manufacturing system

To realize high-accuracy physical-cyber digital twin (DT) mapping in a manufacturing system, a huge amount of data need to be collected and analyzed in real-time. Traditional DTs systems are deployed in cloud or edge servers independently, whilst it is hard to apply in real production systems due to the high interaction or execution delay. This results in a low consistency in the temporal dimension of the physical-cyber model. In this work, we propose a novel efficient edge-cloud DT manufacturing system, which is inspired by resource scheduling technology. Specifically, an edge-cloud collaborative DTs system deployment architecture is first constructed. Then, deterministic and uncertainty optimization adaptive strategies are presented to choose a more powerful server for running DT-based applications. We model the adaptive optimization problems as dynamic programming problems and propose a novel collaborative clustering parallel Q-learning (CCPQL) algorithm and prediction-based CCPQL to solve the problems. The proposed approach reduces the total delay with a higher convergence rate. Numerical simulation results are provided to validate the approach, which would have great potential in dynamic and complex industrial internet environments.

A two-stage scheduler based on New Caledonian Crow Learning Algorithm and reinforcement learning strategy for cloud environment

Most studies on task scheduling in the cloud consider at most two or three objectives and hence the main motivation of this paper is to design the task scheduling problem by considering conflicting objectives (i.e., makespan, energy consumption, resources utilization, and security). The proposed algorithm consists of two stages (i.e., meta-scheduler and local-scheduler). In the meta-scheduler stage, the tasks are assigned to hosts based on their priorities, deadlines, and the power of hosts. In the local-scheduler stage, the optimal mapping between tasks and virtual machines is obtained with the proposed Parallel Reinforcement Learning Caledonian Crow (PRLCC). The proposed PRLCC is a combination of the New Caledonian Crow Learning Algorithm (NCCLA), Reinforcement Learning (RL), and parallel strategy. The RE is utilized to guide the agent's activity and provide a balance between intensification and diversification activities, whereas parallel strategy is employed to help agents for searching different directions of problems in the shortest time. The first experiment tries to evaluate the proposed PRLCC as a global optimizer by 20 test functions (i.e., 8 unimodal and 12 multimodal). The results demonstrate the PRLCC's robustness, efficiency, and stability. The second experiment compares the performance of the proposed scheduler with four scheduling algorithms. In a heavily (lightly) loaded system, it improves waiting time by 32.5% (1.4%), energy consumption by 81% (75%), and resource utilization by 7.5% (3.5%) on average compared to other methods, also guaranteed security by 65.5% (84%).

Joint Beam Training and Data Transmission Control for mmWave Delay-Sensitive Communications: A Parallel Reinforcement Learning Approach

Future communication networks call for new solutions to support their capacity and delay demands by leveraging potentials of the millimeter wave (mmWave) frequency band. However, the beam training procedure in mmWave systems incurs significant overhead as well as huge energy consumption. As such, deriving an adaptive control policy is beneficial to both delay-sensitive and energy-efficient data transmission over mmWave networks. To this end, we investigate the problem of joint beam training and data transmission control for mmWave delay-sensitive communications in this paper. Specifically, the considered problem is firstly formulated as a constrained Markov Decision Process (MDP), which aims to minimize the cumulative energy consumption over the whole considered period of time under delay constraint. By introducing a Lagrange multiplier, we transform the constrained MDP into an unconstrained one, which is then solved via a parallel-rollout-based reinforcement learning method in a data-driven manner. Our numerical results demonstrate that the optimized policy via parallel-rollout significantly outperforms other baseline policies in both energy consumption and delay performance.

A Parallel Deep Reinforcement Learning Framework for Controlling Industrial Assembly Lines

Decision-making in a complex, dynamic, interconnected, and data-intensive industrial environment can be improved with the assistance of machine-learning techniques. In this work, a complex instance of industrial assembly line control is formalized and a parallel deep reinforcement learning approach is presented. We consider an assembly line control problem in which a set of tasks (e.g., vehicle assembly tasks) needs to be planned and controlled during their execution, with the aim of optimizing given key performance criteria. Specifically, the aim will be that of planning the task in order to minimize the total time taken to execute all the tasks (also called cycle time). Tasks run on workstations in the assembly line. To run, tasks need specific resources. Therefore, the tackled problem is that of optimally mapping tasks and resources to workstations, and deciding the optimal execution times of the tasks. In doing so, several constraints need to be respected (e.g., precedence constraints among the tasks, constraints on needed resources to run tasks, deadlines, etc.). The proposed approach uses deep reinforcement learning to learn a tasks/resources mapping policy that is effective in minimizing the resulting cycle time. The proposed method allows us to explicitly take into account all the constraints, and, once training is complete, can be used in real time to dynamically control the execution of tasks. Another motivation for the proposed work is in the ability of the used method to also work in complex scenarios, and in the presence of uncertainties. As a matter of fact, the use of deep neural networks allows for learning the model of the assembly line problem, in contrast with, e.g., optimization-based techniques, which require explicitly writing all the equations of the model of the problem. In order to speed up the training phase, we adopt a learning scheme in which more agents are trained in parallel. Simulations show that the proposed method can provide effective real-time decision support to industrial operators for scheduling and rescheduling activities, achieving the goal of minimizing the total tasks’ execution time.

Embracing Complexity: Agent-Based Modeling for HetNets Design and Optimization via Concurrent Reinforcement Learning Algorithms

Complexity is an inherent property in wireless heterogeneous networks (HetNets). In this paper, we investigate the application of the agent-based modeling (ABM) tool for optimization of complex and dynamic HetNets. The proposed framework contains a diversity of game-theoretic, machine learning, and rule-based algorithms within the same model. We present and analyze a HetNet ABM model that runs parallel reinforcement learning (RL) algorithms for spectrum deployment, interference management, resource allocation, and load balancing at both micro and macrocell levels. In our proposed model, two RL-based algorithms work jointly to manage the co-tier and cross-tier interferences. The macrocell runs the first algorithm to control the transmission power of the small cells. The second RL algorithm is run by small cells to assign the users to the sub-bands with less interference levels. Simultaneously, the user association is decided by the users depending on the available resources at the cells and user preferences. The model is then evaluated under various network load conditions to deduce relationships between the cell loads, aggregate bit rate, latency, and user association. Moreover, the system is assessed in a dynamic network scenario with moving users and is confirmed to possess the ability to attain convergence with sufficient performance levels.

Optimal Beam Association for High Mobility mmWave Vehicular Networks: Lightweight Parallel Reinforcement Learning Approach

In intelligent transportation systems (ITS), vehicles are expected to feature with advanced applications and services which demand ultra-high data rates and low-latency communications. For that, the millimeter wave (mmWave) communication has been emerging as a very promising solution. However, incorporating the mmWave into ITS is particularly challenging due to the high mobility of vehicles and the inherent sensitivity of mmWave beams to dynamic blockages. This article addresses these problems by developing an optimal beam association framework for mmWave vehicular networks under high mobility. Specifically, we use the semi-Markov decision process to capture the dynamics and uncertainty of the environment. The Q-learning algorithm is then often used to find the optimal policy. However, Q-learning is notorious for its slow-convergence. Instead of adopting deep reinforcement learning structures (like most works in the literature), we leverage the fact that there are usually multiple vehicles on the road to speed up the learning process. To that end, we develop a lightweight yet very effective parallel Q-learning algorithm to quickly obtain the optimal policy by simultaneously learning from various vehicles. Extensive simulations demonstrate that our proposed solution can increase the data rate by 47% and reduce the disconnection probability by 29% compared to other solutions.

An Efficient Parallel Reinforcement Learning Approach to Cross-Layer Defense Mechanism in Industrial Control Systems

The ongoing digitalization enables stable control processes and smooth operations of Industrial Control Systems (ICSs). A direct consequence of the highly interconnected architecture of ICSs is the introduced cyber vulnerability and increasing cyber security threats to ICSs. Numerous researches pay attention to the security problem of ICSs. However, most current researches face two challenges. First, the interaction problem between cyber layer and physical layer of ICSs may result incorrect attack response strategies. Second, ICSs are real-time systems, but existing defense decision algorithms based on game theory or reinforcement learning techniques have high computational complexity, which prevents it from making decisions quickly. In this paper, we design a new multi-attribute based reward quantitative method and propose a multi-attribute based Q-learning algorithm to resolve the interaction problem. In addition, to overcome the limitation of slow convergence, we develop an effective parallel Q-learning (PQL) algorithm to quickly find the optimal strategy. The experimental results show the effectiveness of the PQL algorithm. Compared with the Q-learning algorithm (QL) and the deep Q-network (DQN) algorithm, our proposed solution can reduce the average completion time by 12.5%-37%.

Toward a Computational Neuropsychology of Cognitive Flexibility.

Cognitive inflexibility is a well-documented, yet non-specific corollary of many neurological diseases. Computational modeling of covert cognitive processes supporting cognitive flexibility may provide progress toward nosologically specific aspects of cognitive inflexibility. We review computational models of the Wisconsin Card Sorting Test (WCST), which represents a gold standard for the clinical assessment of cognitive flexibility. A parallel reinforcement-learning (RL) model provides the best conceptualization of individual trial-by-trial WCST responses among all models considered. Clinical applications of the parallel RL model suggest that patients with Parkinson’s disease (PD) and patients with amyotrophic lateral sclerosis (ALS) share a non-specific covert cognitive symptom: bradyphrenia. Impaired stimulus-response learning appears to occur specifically in patients with PD, whereas haphazard responding seems to occur specifically in patients with ALS. Computational modeling hence possesses the potential to reveal nosologically specific profiles of covert cognitive symptoms, which remain undetectable by traditionally applied behavioral methods. The present review exemplifies how computational neuropsychology may advance the assessment of cognitive flexibility. We discuss implications for neuropsychological assessment and directions for future research.

Parallel model-based and model-free reinforcement learning for card sorting performance

The Wisconsin Card Sorting Test (WCST) is considered a gold standard for the assessment of cognitive flexibility. On the WCST, repeating a sorting category following negative feedback is typically treated as indicating reduced cognitive flexibility. Therefore such responses are referred to as ‘perseveration’ errors. Recent research suggests that the propensity for perseveration errors is modulated by response demands: They occur less frequently when their commitment repeats the previously executed response. Here, we propose parallel reinforcement-learning models of card sorting performance, which assume that card sorting performance can be conceptualized as resulting from model-free reinforcement learning at the level of responses that occurs in parallel with model-based reinforcement learning at the categorical level. We compared parallel reinforcement-learning models with purely model-based reinforcement learning, and with the state-of-the-art attentional-updating model. We analyzed data from 375 participants who completed a computerized WCST. Parallel reinforcement-learning models showed best predictive accuracies for the majority of participants. Only parallel reinforcement-learning models accounted for the modulation of perseveration propensity by response demands. In conclusion, parallel reinforcement-learning models provide a new theoretical perspective on card sorting and it offers a suitable framework for discerning individual differences in latent processes that subserve behavioral flexibility.

A Computational Study of Executive Dysfunction in Amyotrophic Lateral Sclerosis.

Executive dysfunction is a well-documented, yet nonspecific corollary of various neurological diseases and psychiatric disorders. Here, we applied computational modeling of latent cognition for executive control in amyotrophic lateral sclerosis (ALS) patients. We utilized a parallel reinforcement learning model of trial-by-trial Wisconsin Card Sorting Test (WCST) behavior. Eighteen ALS patients and 21 matched healthy control participants were assessed on a computerized variant of the WCST (cWCST). ALS patients showed latent cognitive symptoms, which can be characterized as bradyphrenia and haphazard responding. A comparison with results from a recent computational Parkinson’s disease (PD) study (Steinke et al., 2020, J Clin Med) suggests that bradyphrenia represents a disease-nonspecific latent cognitive symptom of ALS and PD patients alike. Haphazard responding seems to be a disease-specific latent cognitive symptom of ALS, whereas impaired stimulus-response learning seems to be a disease-specific latent cognitive symptom of PD. These data were obtained from the careful modeling of trial-by-trial behavior on the cWCST, and they suggest that computational cognitive neuropsychology provides nosologically specific indicators of latent facets of executive dysfunction in ALS (and PD) patients, which remain undiscoverable for traditional behavioral cognitive neuropsychology. We discuss implications for neuropsychological assessment, and we discuss opportunities for confirmatory computational brain imaging studies.

Automatic Virtual Network Embedding: A Deep Reinforcement Learning Approach With Graph Convolutional Networks

Virtual network embedding arranges virtual network services onto substrate network components. The performance of embedding algorithms determines the effectiveness and efficiency of a virtualized network, making it a critical part of the network virtualization technology. To achieve better performance, the algorithm needs to automatically detect the network status which is complicated and changes in a time-varying manner, and to dynamically provide solutions that can best fit the current network status. However, most existing algorithms fail to provide automatic embedding solutions in an acceptable running time. In this paper, we combine deep reinforcement learning with a novel neural network structure based on graph convolutional networks, and propose a new and efficient algorithm for automatic virtual network embedding. In addition, a parallel reinforcement learning framework is used in training along with a newly-designed multi-objective reward function, which has proven beneficial to the proposed algorithm for automatic embedding of virtual networks. Extensive simulation results under different scenarios show that our algorithm achieves best performance on most metrics compared with the existing state-of-the-art solutions, with upto 39.6% and 70.6% improvement on acceptance ratio and average revenue, respectively. Moreover, the results also demonstrate that the proposed solution possesses good robustness.

PMA-DRL: A parallel model-augmented framework for deep reinforcement learning algorithms

In recent years, aided by deep neural networks (DNNs), reinforcement learning (RL) algorithms have been achieving great success in more and more tasks. In general, model-free RL algorithms are widely applicable, but sometimes suffer from low sample efficiency. Although the environment dynamics can be incorporated into model-free RL algorithms to enhance sample efficiency, which can be transformed into the joint algorithm of model-free and model-based RL algorithms, the model bias of dynamics model may still hurt the performance. Another attractive study direction for RL algorithms, is using parallel strategy. Meanwhile, parallel RL algorithms can achieve outstanding results under less time cost, but are still with low sample efficiency, because most of such algorithms are vanilla model-free ones. Therefore, aiming to enhance the model performance for RL algorithms with DNNs, in this paper, we propose a novel parallel model-augmented framework, called PMA-DRL, to combine the dynamics model and parallel RL algorithms together. First, we introduce a local dynamics model (LDM) for each local agent model (LAM) in parallel RL algorithms. Next, we propose to build a better LDM by using an ensemble of LDMs, and such ensemble can help to reduce model bias, because the experienced observations under different LAMs are always diverse. Then, the observation diversity can be further increased by employing LDMs to explore. Finally, we implement experiments on classic control tasks (CCTs) and Atari games, which demonstrates the efficiency of our proposed framework.

Parallel reinforcement learning-based energy efficiency improvement for a cyber-physical system

As a complex and critical cyber-physical system &#x0028 CPS &#x0029 , the hybrid electric powertrain is significant to mitigate air pollution and improve fuel economy. Energy management strategy &#x0028 EMS &#x0029 is playing a key role to improve the energy efficiency of this CPS. This paper presents a novel bidirectional long short-term memory &#x0028 LSTM &#x0029 network based parallel reinforcement learning &#x0028 PRL &#x0029 approach to construct EMS for a hybrid tracked vehicle &#x0028 HTV &#x0029 . This method contains two levels. The high-level establishes a parallel system first, which includes a real powertrain system and an artificial system. Then, the synthesized data from this parallel system is trained by a bidirectional LSTM network. The lower-level determines the optimal EMS using the trained action state function in the model-free reinforcement learning &#x0028 RL &#x0029 framework. PRL is a fully data-driven and learning-enabled approach that does not depend on any prediction and predefined rules. Finally, real vehicle testing is implemented and relevant experiment data is collected and calibrated. Experimental results validate that the proposed EMS can achieve considerable energy efficiency improvement by comparing with the conventional RL approach and deep RL.

Parallel Reinforcement Learning With Minimal Communication Overhead for IoT Environments

Many Internet of Things (IoT) applications require a distributed architecture for decision making either because of a lack of a centralized system, failure-prone connectivity to a centralized system or because the imposed latency to contact such a system is too high for real-time applications. Often, these IoT applications fall in the domain of reinforcement learning (RL), e.g., autonomous robot navigation in smart factories and traffic signal control in smart cities. However, RL-based applications require a long learning time. To overcome this limitation and scale with the number of agents, parallel RL (PRL) algorithms run multiple RL agents in parallel and on distributed environments. However, deploying PRL algorithms in such environments entails a communication overhead that increases the (actual) execution time. The state-of-the-art PRL algorithms are designed for reducing the learning time while assuming no (or limited) communication overhead. In this article, we present a novel partitioning algorithm that minimizes the communication overhead in PRL running on IoT environments. To the best of our knowledge, this is the first work that focuses on solving the communication overhead of distributing PRL algorithms without requiring any a priori knowledge about the structure of the problem. The proposed algorithm intelligently combines a dynamic state partitioning strategy, which exploits the agent’s exploration capabilities to build partition knowledge while learning, with an efficient mapping of agents to partitions, which reduces the communication among agents. Performance evaluations show that the proposed algorithm can achieve almost no communication among PRL agents at the converged state.