Quantum Reinforcement Learning Research Articles

Automatic evolutionary design of quantum rule-based systems and applications to quantum reinforcement learning

Explainable artificial intelligence is a research topic whose relevance has increased in recent years, especially with the advent of large machine learning models. However, very few attempts have been proposed to improve interpretability in the case of quantum artificial intelligence, and many existing quantum machine learning models in the literature can be considered almost as black boxes. In this article, we argue that an appropriate semantic interpretation of a given quantum circuit that solves a problem can be of interest to the user not only to certify the correct behavior of the learned model, but also to obtain a deeper insight into the problem at hand and its solution. We focus on decision-making problems that can be formulated as classification tasks and propose a method for learning quantum rule-based systems to solve them using evolutionary optimization algorithms. The approach is tested to learn rules that solve control and decision-making tasks in reinforcement learning environments, to provide interpretable agent policies that help to understand the internal dynamics of an unknown environment. Our results conclude that the learned policies are not only highly explainable, but can also help detect non-relevant features of problems and produce a minimal set of rules.

Brain-Inspired Agents for Quantum Reinforcement Learning

Brain-Inspired Agents for Quantum Reinforcement Learning

Quafu-RL: The cloud quantum computers based quantum reinforcement learning

With the rapid advancement of quantum computing, hybrid quantum–classical machine learning has shown numerous potential applications at the current stage, with expectations of being achievable in the noisy intermediate-scale quantum (NISQ) era. Quantum reinforcement learning, as an indispensable study, has recently demonstrated its ability to solve standard benchmark environments with formally provable theoretical advantages over classical counterparts. However, despite the progress of quantum processors and the emergence of quantum computing clouds, implementing quantum reinforcement learning algorithms utilizing parameterized quantum circuits (PQCs) on NISQ devices remains infrequent. In this work, we take the first step towards executing benchmark quantum reinforcement problems on real devices equipped with at most 136 qubits on the BAQIS Quafu quantum computing cloud. The experimental results demonstrate that the policy agents can successfully accomplish objectives under modified conditions in both the training and inference phases. Moreover, we design hardware-efficient PQC architectures in the quantum model using a multi-objective evolutionary algorithm and develop a learning algorithm that is adaptable to quantum devices. We hope that the Quafu-RL can be a guiding example to show how to realize machine learning tasks by taking advantage of quantum computers on the quantum cloud platform.

Efficient relation extraction via quantum reinforcement learning

Most existing relation extraction methods only determine the relation type after identifying all entities, thus not fully modeling the interaction between relation-type recognition and entity mention detection. This article introduces a novel paradigm for relation extraction by treating relevant entities as parameters of relations and harnessing the strong expressive capabilities and acceleration advantages of quantum computing to address the relation extraction task. In this article, we develop a quantum hierarchical reinforcement learning approach to enhance the interaction between relation-type recognition and entity mention detection. The entire relation extraction process is broken down into a hierarchical structure of two layers of quantum reinforcement learning strategies dedicated to relation detection and entity extraction, demonstrating greater feasibility and expressiveness, especially when dealing with superimposed relations. Our proposed method outperforms existing approaches through experimental evaluations on commonly used public datasets, mainly showcasing its significant advantages in extracting superimposed relationships.

Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines

Free energy-based reinforcement learning (FERL) with clamped quantum Boltzmann machines (QBM) was shown to significantly improve the learning efficiency compared to classical Q-learning with the restriction, however, to discrete state-action space environments. In this paper, the FERL approach is extended to multi-dimensional continuous state-action space environments to open the doors for a broader range of real-world applications. First, free energy-based Q-learning is studied for discrete action spaces, but continuous state spaces and the impact of experience replay on sample efficiency is assessed. In a second step, a hybrid actor-critic (A-C) scheme for continuous state-action spaces is developed based on the deep deterministic policy gradient algorithm combining a classical actor network with a QBM-based critic. The results obtained with quantum annealing (QA), both simulated and with D-Wave QA hardware, are discussed, and the performance is compared to classical reinforcement learning methods. The environments used throughout represent existing particle accelerator beam lines at the European Organisation for Nuclear Research. Among others, the hybrid A-C agent is evaluated on the actual electron beam line of the Advanced Wakefield Experiment (AWAKE).

Analysis on quantum reinforcement learning algorithms for prediction of protein sequence

Analysis on quantum reinforcement learning algorithms for prediction of protein sequence

Quantum Reinforcement Learning for Spatio-Temporal Prioritization in Metaverse

Quantum Reinforcement Learning for Spatio-Temporal Prioritization in Metaverse

Realizing Stabilized Landing for Computation-Limited Reusable Rockets: A Quantum Reinforcement Learning Approach

Realizing Stabilized Landing for Computation-Limited Reusable Rockets: A Quantum Reinforcement Learning Approach

Quantum Deep Hedging

Quantum machine learning has the potential for a transformative impact across industry sectors and in particular in finance. In our work we look at the problem of hedging where deep reinforcement learning offers a powerful framework for real markets. We develop quantum reinforcement learning methods based on policy-search and distributional actor-critic algorithms that use quantum neural network architectures with orthogonal and compound layers for the policy and value functions. We prove that the quantum neural networks we use are trainable, and we perform extensive simulations that show that quantum models can reduce the number of trainable parameters while achieving comparable performance and that the distributional approach obtains better performance than other standard approaches, both classical and quantum. We successfully implement the proposed models on a trapped-ion quantum processor, utilizing circuits with up to 16 qubits, and observe performance that agrees well with noiseless simulation. Our quantum techniques are general and can be applied to other reinforcement learning problems beyond hedging.

Quantum reinforcement learning in the presence of thermal dissipation.

A study of the effect of thermal dissipation on quantum reinforcement learning is performed. For this purpose, a nondissipative quantum reinforcement learning protocol is adapted to the presence of thermal dissipation. Analytical calculations as well as numerical simulations are carried out, obtaining evidence that dissipation does not significantly degrade the performance of the quantum reinforcement learning protocol for sufficiently low temperatures, in some cases even being beneficial. Quantum reinforcement learning under realistic experimental conditions of thermal dissipation opens an avenue for the realization of quantum agents to be able to interact with a changing environment, as well as adapt to it, with many plausible applications inside quantum technologies and machine learning.

Quantum reinforcement learning via policy iteration

Quantum reinforcement learning via policy iteration

Secure Communication Based On Key Generation With Quantum Reinforcement Learning

Data security and secure communication is one of the most important issues of today. In this study, a quantum-based
 method for secure communication is proposed. In the proposed method, the necessary secret key in communication is generated
 locally by each participant through quantum gates. The quantum gates are taught by using quantum reinforcement learning (QRL).
 Proposed study is simulated using the Qiskit library for Python. Proposed study performs the learning action with an accuracy of
 87.95% for 195 gates, 85.47% for 128 gates, 83.59% for 64 gates, 76.25% for 32 gates. As the key size increases, the performance
 of the method increases. The participants don’t share the secret key in the presented method. Thus, the communication becomes
 more secure. In the study, the method is also examined in terms of security. Security analysis shows that the proposed method
 provide secure communication.

Offline Quantum Reinforcement Learning in a Conservative Manner

Recently, to reap the quantum advantage, empowering reinforcement learning (RL) with quantum computing has attracted much attention, which is dubbed as quantum RL (QRL). However, current QRL algorithms employ an online learning scheme, i.e., the policy that is run on a quantum computer needs to interact with the environment to collect experiences, which could be expensive and dangerous for practical applications. In this paper, we aim to solve this problem in an offline learning manner. To be more specific, we develop the first offline quantum RL (offline QRL) algorithm named CQ2L (Conservative Quantum Q-learning), which learns from offline samples and does not require any interaction with the environment. CQ2L utilizes variational quantum circuits (VQCs), which are improved with data re-uploading and scaling parameters, to represent Q-value functions of agents. To suppress the overestimation of Q-values resulting from offline data, we first employ a double Q-learning framework to reduce the overestimation bias; then a penalty term that encourages generating conservative Q-values is designed. We conduct abundant experiments to demonstrate that the proposed method CQ2L can successfully solve offline QRL tasks that the online counterpart could not.

Quantum Machine Learning Implementations: Proposals and Experiments

AbstractThis article gives an overview and a perspective of recent theoretical proposals and their experimental implementations in the field of quantum machine learning. Without an aim to being exhaustive, the article reviews specific high‐impact topics such as quantum reinforcement learning, quantum autoencoders, and quantum memristors, and their experimental realizations in the platforms of quantum photonics and superconducting circuits. The field of quantum machine learning can be among the first quantum technologies producing results that are beneficial for industry and, in turn, to society. Therefore, it is necessary to push forward initial quantum implementations of this technology, in noisy intermediate‐scale quantum computers, aiming for achieving fruitful calculations in machine learning that are better than with any other current or future computing paradigm.

Blockchain Based Optimized Energy Trading for E-Mobility Using Quantum Reinforcement Learning

Using microgrids to charge Electric Vehicles (EVs) is a significant step toward achieving Electric mobility. The microgrids generate electricity for self-use and sell surplus energy locally in Peer-to-Peer (P2P) manner, where seller and buyer meet to trade electricity directly on agreed terms without any intermediary. The energy trading decision for the microgrid is a challenging issue due to uncertainty of renewable energy yield, the electric demands of EVs and without knowing the offers of other competitive microgrids together makes it hard to decide the selling price of energy per unit. Further, there is a need to audit and verify the energy trading and store energy transactions securely in distributed manner to avoid collapse of the system in case of single point of failure. In this context, this paper presents a Blockchain and Quantum Reinforcement Learning based optimized Energy Trading (BQL-ET) model for E-mobility. Firstly, a double-auction mechanism is proposed to set optimal market-trading price by observing the selling price of each microgrid and the demand of EV's. Secondly, using smart contracts, consortium blockchain is deployed for the evaluation of overall utility, which includes energy supply, demand, and cost for both microgrids and EVs. Finally, Utility maximization problem is transformed into a Markov Decision Process (MDP), and in order to develop the learning policy and maximize overall utility, a QRL optimization for solving the MDP problem is proposed. Convergence analysis and performance results attest that BQL-ET convergences faster, maximizes the utility of both microgrids an EVs with lower transaction confirmation time and setting of the optimal market-trading price compared to state-of-the art models.

Robustness of quantum reinforcement learning under hardware errors

Variational quantum machine learning algorithms have become the focus of recent research on how to utilize near-term quantum devices for machine learning tasks. They are considered suitable for this as the circuits that are run can be tailored to the device, and a big part of the computation is delegated to the classical optimizer. It has also been hypothesized that they may be more robust to hardware noise than conventional algorithms due to their hybrid nature. However, the effect of training quantum machine learning models under the influence of hardware-induced noise has not yet been extensively studied. In this work, we address this question for a specific type of learning, namely variational reinforcement learning, by studying its performance in the presence of various noise sources: shot noise, coherent and incoherent errors. We analytically and empirically investigate how the presence of noise during training and evaluation of variational quantum reinforcement learning algorithms affect the performance of the agents and robustness of the learned policies. Furthermore, we provide a method to reduce the number of measurements required to train Q-learning agents, using the inherent structure of the algorithm.

Quantum reinforcement learning

In this paper, we present implementations of an annealing-based and a gate-based quantum computing approach for finding the optimal policy to traverse a grid and compare them to a classical deep reinforcement learning approach. We extended these three approaches by allowing for stochastic actions instead of deterministic actions and by introducing a new learning technique called curriculum learning. With curriculum learning, we gradually increase the complexity of the environment and we find that it has a positive effect on the expected reward of a traversal. We see that the number of training steps needed for the two quantum approaches is lower than that needed for the classical approach.

Quantum Reinforcement Learning: Analysis of the Influence of the Pqc Architecture, Q-Values and Observables on Agent Learning and Performance

Quantum Reinforcement Learning: Analysis of the Influence of the Pqc Architecture, Q-Values and Observables on Agent Learning and Performance

Asynchronous training of quantum reinforcement learning

The development of quantum machine learning (QML) has received a lot of interest recently thanks to developments in both quantum computing (QC) and machine learning (ML). One of the ML paradigms that can be utilized to address challenging sequential decision-making issues is reinforcement learning (RL). It has been demonstrated that classical RL can successfully complete many difficult tasks. A leading method of building quantum RL agents relies on the variational quantum circuits (VQC). However, training QRL algorithms with VQCs requires significant amount of computational resources. This issue hurdles the exploration of various QRL applications. In this paper, we approach this challenge through asynchronous training of QRL agents. Specifically, we choose the asynchronous training of advantage actor-critic variational quantum policies. We demonstrate the results via numerical simulations that within the tasks considered, the asynchronous training of QRL agents can reach performance comparable to or superior to classical agents with similar model sizes and architectures.

NFT-Based Intelligence Networking for Connected and Autonomous Vehicles: A Quantum Reinforcement Learning Approach

Recently, the Internet of vehicles (IoV) and connected and autonomous vehicles (CAVs) have become research hotspots. The accuracy and efficiency of artificial intelligence (AI) models are crucial for CAVs to make decisions automatically. Intelligence networking enables each CAV to train appropriate models locally with the help of other CAVs' intelligence and make up for the lack of experience and computing power of a single-vehicle. However, intelligence is distributed across diverse geo-locations in the intelligence networking paradigm, which calls for accurate, efficient, and lightweight intelligence discovery mechanisms. In this article, we propose a non-fungible token (NFT)-based distributed intelligence networking scheme (NDIN) for CAVs. We use NFT to tokenize intelligence and efficiently describe intelligence from multiple aspects through meta-data, facilitating applications to better understand and search for intelligence in complex and trust-lacking IoV. We present the architecture, modules, and core mechanism of NDIN. Then, we formulate the essential problem as a discrete Markov decision process (MDP) and adopt the quantum-inspired reinforcement learning (QRL) algorithm to find the optimal policy. Also, the convergence rate and performance compared with existing schemes are evaluated. Finally, we discuss several related challenges and opportunities.