Large State Space Research Articles

Cooperative Task Offloading for Mobile Edge Computing Based on Multi-Agent Deep Reinforcement Learning

Driven by the prevalence of the computation-intensive and delay-intensive mobile applications, Mobile Edge Computing (MEC) is emerging as a promising solution. Traditional task offloading methods usually rely on centralized decision making, which inevitably involves a high computational complexity and a large state space. However, the MEC is a typical distributed system, where the edge servers are geographically separated, and independently perform the computing tasks. This fact inspires us to conceive a distributed cooperative task offloading system, where each edge server makes its own decision on how to allocate local computing resources and how to migrate tasks among the edge servers. To characterize diverse task requirements, we divide the arrival tasks into different priorities according to the tolerance time, which enables to dynamically schedule the local computing resources for reducing the task timeout. In order to coordinate the independent decision makings of geographically separate edge servers, we propose a priority driven cooperative task offloading algorithm based on multi-agent deep reinforcement learning, where the decision making of each edge server not only depends on its own state but also on the shared global information. We further develop a Variational Recurrent Neural Network (VRNN) based global state sharing model which significantly reduces the communication overhead among edge servers. The performance evaluation conducted on a movement trajectories dataset of mobile devices verifies that the proposed algorithm can reduce the task consumption time and improve the edge computing resources utilization.

Markov decision processes with burstiness constraints

We consider a Markov Decision Process (MDP), over a finite or infinite horizon, augmented by so-called (σ,ρ)-burstiness constraints. Such constraints, which had been introduced within the framework of network calculus, are meant to limit some additive quantity to a given rate over any time interval, plus a term which allows for occasional and limited bursts. We introduce this class of constraints for MDP models, and formulate the corresponding constrained optimization problems. Due to the burstiness constraints, constrained optimal policies are generally history-dependent. We use a recursive form of the constraints to define an augmented-state model, for which sufficiency of Markov or stationary policies is recovered and the standard theory may be applied, albeit over a larger state space. The analysis is mainly devoted to a characterization of feasible policies, followed by application to the constrained MDP optimization problem. A simple queuing example serves to illustrate some of the concepts and calculations involved.

: A Semantics-Guided Safety Enhancement Framework for AI-Enabled Cyber-Physical Systems

Cyber-Physical Systems (CPSs) have been widely adopted in various industry domains to support many important tasks that impact our daily lives, such as automotive vehicles, robotics manufacturing, and energy systems. As Artificial Intelligence (AI) has demonstrated its promising abilities in diverse tasks like decision-making, prediction, and optimization, a growing number of CPSs adopt AI components in the loop to further extend their efficiency and performance. However, these modern AI-enabled CPSs have to tackle pivotal problems that the AI-enabled control systems might need to compensate the balance across <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multiple operation requirements</i> and avoid possible defections in advance to safeguard human lives and properties. Modular redundancy and ensemble method are two widely adopted solutions in the traditional CPSs and AI communities to enhance the functionality and flexibility of a system. Nevertheless, there is a lack of deep understanding of the effectiveness of such ensemble design on AI-CPSs across diverse industrial applications. Considering the complexity of AI-CPSs, existing ensemble methods fall short of handling such huge state space and sophisticated system dynamics. Furthermore, an ideal control solution should consider the multiple system specifications in real-time and avoid erroneous behaviors beforehand. Such that, a new specification-oriented ensemble control system is of urgent need for AI-CPSs. In this paper, we propose <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathtt {SIEGE}$</tex-math></inline-formula> , a semantics-guided ensemble control framework to initiate an early exploratory study of ensemble methods on AI-CPSs and aim to construct an efficient, robust, and reliable control solution for multi-tasks AI-CPSs. We first utilize a semantic-based abstraction to decompose the large state space, capture the ongoing system status and predict future conditions in terms of the satisfaction of specifications. We propose a series of new semantics-aware ensemble strategies and an end-to-end Deep Reinforcement Learning (DRL) hierarchical ensemble method to improve the flexibility and reliability of the control systems. Our large-scale, comprehensive evaluations over five subject CPSs show that 1) the semantics abstraction can efficiently narrow the large state space and predict the semantics of incoming states, 2) our semantics-guided methods outperform state-of-the-art individual controllers and traditional ensemble methods, and 3) the DRL hierarchical ensemble approach shows promising capabilities to deliver a more robust, efficient, and safety-assured control system. To enable further research along this direction to build better AI-enabled CPS, we made all of the code and experimental results data publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://sites.google.com/view/ai-cps-siege/home</uri> .

DQN-ALrM-Based Intelligent Handover Method for Satellite-Ground Integrated Network

The satellite-ground integrated network (SGIN), consisting of satellites and ground base stations, is regarded as the trend to solve the problem of global coverage. However, highly dynamic scenarios pose great challenges to users’ mobility management, especially handover decisions. With the rapid development of machine learning, reinforcement learning (RL), such as Q-learning and deep Q-network (DQN), has been applied to handover decision making owing to being good at solving dynamic multi-attribute optimization problems. However, Q-learning can be a waste of storage for large state spaces. Besides, the existing algorithms for training a neural network in a deep RL framework have problems of slow convergence or unstable convergence performance. Therefore, we propose a handover method exploiting the DQN framework of adaptive learning rate with momentum (DQN-ALrM), which can not only improve decision accuracy, but also improve learning efficiency. The customized DQN framework can solve problems with a large-size state space, and the proposed ALrM method can adjust the learning rate at any time according to the training error situation. Comprehensive simulations are conducted to validate the superiority of the proposed method in convergence speed, handover rate, call failure rate and multi-metric quality of service (QoS) over the alternative solutions in the literature.

Capacity allocation in service systems with preferred delivery times and multiple customer classes

Motivated by operational problems in click-and-collect systems, such as curbside pickup programs, we study a joint admission control and capacity allocation problem. We consider systems where customers have preferred service delivery times and can be of different priority classes. The service provider can reject customers upon arrival or serve jobs via overtime when service capacity is insufficient. The service provider’s goal is to find the minimum-cost admission and capacity allocation policy to dynamically decide when to serve and whom to serve. We model this problem as a Markov Decision Process and present structural results to partially characterize suboptimal solutions. We then develop a linear programming-based exact solution method using these results. We also present a problem-specific approximation method using a new state aggregation rule to address computational challenges faced due to large state and action spaces. Finally, we develop heuristic policies for large instances based on the behavior of optimal policies in small problems. We evaluate our methods through extensive computational experiments where we vary the service capacity, arrivals, associated service costs, customer segmentation, and order patterns. Our solution methods perform significantly better than several benchmarks in managing the tradeoff between the computation time and solution quality.

Inference on extended-spectrum beta-lactamase Escherichia coli and Klebsiella pneumoniae data through SMC2

Abstract We propose a novel stochastic model for the spread of antimicrobial-resistant bacteria in a population, together with an efficient algorithm for fitting such a model to sample data. We introduce an individual-based model for the epidemic, with the state of the model determining which individuals are colonised by the bacteria. The transmission rate of the epidemic takes into account both individuals’ locations, individuals’ covariates, seasonality, and environmental effects. The state of our model is only partially observed, with data consisting of test results from individuals from a sample of households. Fitting our model to data is challenging due to the large state space of our model. We develop an efficient SMC2 algorithm to estimate parameters and compare models for the transmission rate. We implement this algorithm in a computationally efficient manner by using the scale invariance properties of the underlying epidemic model. Our motivating application focuses on the dynamics of community-acquired extended-spectrum beta-lactamase-producing Escherichia coli and Klebsiella pneumoniae, using data collected as part of the Drivers of Resistance in Uganda and Malawi project. We infer the parameters of the model and learn key epidemic quantities such as the effective reproduction number, spatial distribution of prevalence, household cluster dynamics, and seasonality.

A topic-aware classifier based on a hybrid quantum-classical model

In the era of Large Language Models, there is still potential for improvement in current Natural Language Processing (NLP) methods in terms of verifiability and consistency. NLP classical approaches are computationally expensive due to their high-power consumption, computing power, and storage requirements. Another computationally efficient approach to NLP is categorical quantum mechanics, which combines grammatical structure and individual word meaning to deduce the sentence meaning. As both quantum theory and natural language use vector space to describe states which are more efficient on quantum hardware, QNLP models can achieve up to quadratic speedup over classical direct calculation methods. In recent years, there is significant progress in utilizing quantum features such as superposition and entanglement to represent linguistic meaning on quantum hardware. Earlier research work has already demonstrated QNLP’s potential quantum advantage in terms of speeding up search, enhancing classification tasks’ accuracy and providing an exponentially large quantum state space in which complex linguistic structures can be efficiently embedded. In this work, a QNLP model is used to determine if two sentences are related to the same topic or not. By comparing our QNLP model to a classical tensor network-based one, our model improved training accuracy by up to 45% and validation accuracy by 35%, respectively. The QNLP model convergence is also studied when varying: first, the problem size, second, parametrized quantum circuits used for model’s training, and last, the backend quantum simulator noise model. The experimental results show that strongly entangled ansatz designs result in fastest model convergence.

Imprecise Probabilistic Model Checking for Stochastic Multi-agent Systems

Standard techniques for model checking stochastic multi-agent systems usually assume the transition probabilities describing the system dynamics to be stationary and completely specified. As a consequence, neither non-stationary systems nor systems whose stochastic behaviour is partially unknown can be treated. So far, most of the approaches proposed to overcome this limitation suffer from complexity issues making them poorly efficient in the case of large state spaces. A fruitful but poorly explored way out is offered by the formalism of imprecise probabilities and the related imprecise Markov models. The aim of this paper is to show how imprecise probabilities can be fruitfully involved to model-check multi-agent systems characterised by non-stationary behaviours. Specifically, the paper introduces a new class of multi-agent models called Imprecise Probabilistic Interpreted Systems and their relative extensions with rewards. It also introduces a proper logical language to specify properties of such models and corresponding model checking algorithms based on iterative procedures to compute probabilistic and epistemic inferences over imprecise Markov models.

Specification-Based Symbolic Execution for Stateful Network Protocol Implementations in IoT

IoT devices offer insufficient protections against exploitation of critical programming errors (such as buffer overflows) it is therefore paramount to sufficiently test IoT software before deployment. A central source of these errors are implementations of stateful network protocols used in the IoT (e.g. MQTT-SN). Unfortunately, comprehensive automated testing of such protocol implementations is challenging due to their large state space. Furthermore, prior work on automated testing of protocol implementations focuses on conventional desktop systems and does thus not address challenges specific to the IoT domain (e.g. handling of peripheral interactions). In this publication, we leverage symbolic execution (an automated software testing technique) for testing stateful network protocol implementation in the IoT domain. To mitigate issues regarding state space explosion, we propose a novel symbolic execution approach which is based on created protocol state and message format specifications and specifically tailored to the IoT domain. We employ our specification-based symbolic execution approach to test protocol implementations provided by two popular IoT operating systems. Our experiments indicate that a significant increase in code coverage can be achieved using our automated software testing approach. We were further able to discover three critical and previously unknown bugs in network protocol implementations provided by the RIOT operating system.

Leveraging UAV-RIS Reflects to Improve the Security Performance of Wireless Network Systems

In this letter, a novel application method of reconfigurable intelligent surfaces (RIS) is proposed, which installs a RIS on an unmanned aerial vehicle (UAV) rather than fixed to buildings. When the location information of the eavesdropper is unknown, the UAV’s rapid deployment and RIS’s ability to change the transmission channel are utilized to dynamically optimize the UAV’s flight trajectory and RIS’s phase shift. The trajectory of the UAV and phase shift of the RIS are optimized to avoid the risk of potential eavesdropping and improve the average downlink secrecy rate of the system. This letter utilizes Q-learning and Deep Q-network (DQN) algorithms to solve the problem. Experiments show that RIS can reduce the flight distance of the UAV on the premise of ensuring security. The scenarios where the two algorithms are applicable are analyzed, and the DQN algorithm is more suitable for the scenarios of large state and action spaces. Also, the experimental results imply that the RIS-assisted UAV improves the average downlink secrecy rate and system security.

Timely Monitoring of Dynamic Sources With Observations From Multiple Wireless Sensors

Age of Information (AoI) has recently received much attention due to its relevance for IoT sensing and monitoring. In this paper, we consider the problem of minimizing the AoI in a system in which a set of sources are observed by multiple sensors in a many-to-many relationship, and the probability that a sensor observes a source depends on the source’s state. This model represents many practical scenarios, such as when multiple cameras or microphones are deployed to monitor objects moving in certain areas. We formulate the scheduling problem as a Markov Decision Process, and show how the age-optimal scheduling policy can be obtained. We further consider partially observable variants of the problem, and devise approximate policies for large state spaces. The evaluations show that the approximate policies work well in the considered scenarios, while the fact that sensors can observe multiple sources is beneficial, especially when there is high uncertainty of the source states.

Imperfect-Information Game AI Agent Based on Reinforcement Learning Using Tree Search and a Deep Neural Network

In the field of computer intelligence, it has always been a challenge to construct an agent model that can be adapted to various complex tasks. In recent years, based on the planning algorithm of Monte Carlo tree search (MCTS), a new idea has been proposed to solve the AI problems of two-player zero-sum games such as chess and Go. However, most of the games in the real environment rely on imperfect information, so it is impossible to directly use the normal tree search planning algorithm to construct a decision-making model. Mahjong, which is a popular multiplayer game with a long history in China, attracts great attention from AI researchers because it contains a large game state space and a lot of hidden information. In this paper, we utilize an agent learning approach that leverages deep learning, reinforcement learning, and dropout learning techniques to implement a Mahjong AI game agent. First, we improve the state transition of the tree search based on the learned MDP model, the player position variable and transition information are introduced into the tree search algorithm to construct a multiplayer search tree. Then, the model training based on a deep reinforcement learning method ensures the stable and sustainable training process of the learned MDP model. Finally, we utilize the strategy data generated by the tree search and use the dropout learning method to train the normal decision-making agent. The experimental results demonstrate the efficiency and stability performance of the agent trained by our proposed method compared with existing agents in terms of test data accuracy, tournament ranking performance, and online match performance. The agent plays against human players and acts like real humans.

Analysing ill-conditioned Markov chains.

Discrete state Markov chains in discrete or continuous time are widely used to model phenomena in the social, physical and life sciences. In many cases, the model can feature a large state space, with extreme differences between the fastest and slowest transition timescales. Analysis of such ill-conditioned models is often intractable with finite precision linear algebra techniques. In this contribution, we propose a solution to this problem, namely partial graph transformation, to iteratively eliminate and renormalize states, producing a low-rank Markov chain from an ill-conditioned initial model. We show that the error induced by this procedure can be minimized by retaining both the renormalized nodes that represent metastable superbasins, and those through which reactive pathways concentrate, i.e. the dividing surface in the discrete state space. This procedure typically returns a much lower rank model, where trajectories can be efficiently generated with kinetic path sampling. We apply this approach to an ill-conditioned Markov chain for a model multi-community system, measuring the accuracy by direct comparison with trajectories and transition statistics. This article is part of a discussion meeting issue 'Supercomputing simulations of advanced materials'.

A Quantum States Preparation Method Based on Difference-Driven Reinforcement Learning

Due to the large state space of the two-qubit system, and the adoption of ladder reward function in the existing quantum state preparation methods, the convergence speed is slow and it is difficult to prepare the desired target quantum state with high fidelity under limited conditions. To solve the above problems, a difference-driven reinforcement learning (RL) algorithm for quantum state preparation of two-qubit system is proposed by improving the reward function and action selection strategy. Firstly, a model is constructed for the problem of preparing quantum states of a two-qubit system, with restrictions on the type of quantum gates and the time for quantum state evolution. In the preparation process, a weighted differential dynamic reward function is designed to assist the algorithm quickly obtain the maximum expected cumulative reward. Then, an adaptive [Formula: see text]-greedy action selection strategy is adopted to achieve a balance between exploration and utilization to a certain extent, thereby improving the fidelity of the final quantum state. The simulation results show that the proposed algorithm can prepare quantum state with high fidelity under limited conditions. Compared with other algorithms, it has different degrees of improvement in convergence speed and fidelity of the final quantum state.

State space geometry of the spin-1 antiferromagnetic Heisenberg chain

We study the phase diagram of the spin-1 antiferromagnetic Heisenberg chain with uniaxial anisotropy and applied magnetic field in terms of the genuine multipartite entanglement as witnessed by the mean quantum Fisher information density. By considering the manifold generated by the staggered magnetization operator, we connect the state space curvature in the vicinity of the ground state of the Heisenberg chain to the genuine multipartite entanglement. Our analysis demonstrates that the quantum critical points and symmetry protected topological (SPT) phase exhibit large state space curvature, while the separable phases are completely flat, offering insight into the physical interpretation of state space curvature. We further show that the entanglement in the SPT phase is enhanced by the presence of uniaxial anisotropy and undiminished in the presence of uniform magnetic fields. The magnon condensate phase induced by large fields is shown to emanate from the Gaussian critical point and exhibits massive multipartite entanglement over a robust region of the parameter space.

Deep reinforcement learning based relaying for buffer-aided cooperative communications

The advances in deep reinforcement learning (DRL) have shown a great potential in solving physical layer-related communication problems. This paper investigates DRL for the relay selection in buffer-aided (BA) cooperative networks. The capability of DRL in handling highly-dimensional problems with large state and action spaces paves the way for exploring additional degrees-of-freedom by relaxing the restrictive assumptions around which conventional cooperative networks are usually designed. This direction is examined in our work by advising and analyzing advanced DRL-based BA relaying strategies that can cope with a variety of setups in multifaceted cooperative networks. In particular, we advise novel BA relaying strategies for both parallel-relaying and serial-relaying systems. For parallel-relaying systems, we investigate the added value of merging packets at the relays and of activating the inter-relay links. For serial-relaying (multi-hop) systems, we explore the improvements that can be reaped by merging packets and by allowing for the simultaneous activation of sufficiently-spaced hops. Simulation results demonstrate the capability of DRL-based BA relaying in achieving substantial improvements in the network throughput while the adequate design of the reward/punishment in the learning process ensures fast convergence speeds.

CenLight: Centralized traffic grid signal optimization via action and state decomposition

AbstractThe centralized traffic grid signal control by the reinforcement learning method is challenging due to the difficulties of searching policy in the large state and action space. In order to solve these problems, a deep reinforcement learning (DRL) method via the action and state decomposition mechanism is proposed. We apply long short‐term memory to construct the agent which decomposes the high‐dimensional state and action space into sub‐spaces and makes decisions incrementally. This is a significant difference between our method and other methods. Through the specifically designed structure of the agent, the difficulty of searching policies can be mitigated, and our method can effectively control the traffic lights in a grid with hundreds of intersections. Experiments on synthetic data and real‐world data show that our method has better performance than traditional control methods and state‐of‐the‐art DRL‐based methods with an improvement of 21% on the queue length on synthetic data and the best travel time with an improvement of 9.35% on real‐world data.

Efficient Distributed Energy Resource Voltage Control Using Ensemble Deep Reinforcement Learning

To meet the challenges of low-carbon power generation, distributed energy resources (DERs) such as solar and wind power generators are now widely integrated into the power grid. Because of the autonomous nature of DERs, ensuring properly regulated output voltages of the individual sources to the grid distribution system poses a technical challenge to grid operators. Stochastic, model-free voltage regulation methods such as deep reinforcement learning (DRL) have proven effective in the regulation of DER output voltages; however, deriving an optimal voltage control policy using DRL over a large state space has a large computational time complexity. In this paper, we illustrate a computationally efficient method for deriving an optimal voltage control policy using a parallelized DRL ensemble. Additionally, we illustrate the resiliency of the control ensemble when random noise is introduced by a cyber-adversary.

Deep reinforce learning for joint optimization of condition-based maintenance and spare ordering

Condition-based maintenance (CBM) policy can avoid premature or late maintenance and reduce system failures and maintenance costs. Most existing CBM studies cannot solve the dimensional disaster problem in multi-component complex systems. Only some studies consider the constraint of maintenance resources when searching for the optimal maintenance policy, which is hard to apply to practical maintenance. This paper studies the joint optimization of the CBM policy and spare components inventory for the multi-component system in large state and action spaces. We use Markov Decision Process to model it and propose an improved deep reinforcement learning algorithm based on the stochastic policy and actor-critic framework. In this algorithm, factorization decomposes the system action into the linear combination of each component's action. The experimental results show that the algorithm proposed in this paper has better time performance and lower system cost compared with other benchmark algorithms. The training time of the former is only 28.5% and 9.12% of that of PPO and DQN algorithms, and the corresponding system cost is decreased by 17.39% and 27.95%, respectively. At the same time, our algorithm has good scalability and is suitable for solving Markov decision-making problems in large-scale state and action space.

Hierarchical reinforcement learning for transportation infrastructure maintenance planning

Maintenance planning on bridges commonly faces multiple challenges, mainly related to complexity and scale. Those challenges stem from the large number of structural elements in each bridge in addition to the uncertainties surrounding their health condition, which is monitored using visual inspections at the element-level. Recent developments have relied on deep reinforcement learning (RL) for solving maintenance planning problems, with the aim to minimize the long-term costs. Nonetheless, existing RL based solutions have adopted approaches that often lacked the capacity to scale due to the inherently large state and action spaces. The aim of this paper is to introduce a hierarchical RL formulation for maintenance planning, which naturally adapts to the hierarchy of information and decisions in infrastructure. The hierarchical formulation enables decomposing large state and action spaces into smaller ones, by relying on state and temporal abstraction. An additional contribution from this paper is the development of an open-source RL environment that uses state-space models (SSM) to describe the propagation of the deterioration condition and speed over time. The functionality of this new environment is demonstrated by solving maintenance planning problems at the element-level, and the bridge-level.