Large State Space Research Articles

Learning Strategies for Real-Time Strategy Games with Genetic Programming

Planning in real-time strategy (RTS) games is challenging due to their very large state and action spaces. Action abstractions have shown to be a promising approach for dealing with this challenge. Previous approaches induce action abstractions from a small set of hand-crafted strategies, which are used by algorithms to search only on the actions returned by the strategies. Previous works use a set of expert-designed strategies for inducing action abstractions. The main drawback of this approach is that it limits the agent behaviour to the knowledge encoded in the strategies. In this research, we focus on learning novel and effective strategies for RTS games, to induce action abstractions. In addition to being effective, we are interested in learning strategies that can be easily interpreted by humans, allowing a better understanding of the workings of the resulting agent.

Dynamic Pricing of Wireless Internet Based on Usage and Stochastically Changing Capacity

Problem definition: Inspired by new developments in dynamic spectrum access, we study the dynamic pricing of wireless Internet access when demand and capacity (bandwidth) are stochastic. Academic/practical relevance: The demand for wireless Internet access has increased enormously. However, the spectrum available to wireless service providers is limited. The industry has, thus, altered conventional license-based spectrum access policies through unlicensed spectrum operations. The additional spectrum obtained through these operations has stochastic capacity. Thus, the pricing of this service by the service provider has novel challenges. The problem considered in this paper is, therefore, of high practical relevance and new to the academic literature. Methodology: We study this pricing problem using a Markov decision process model in which customers are posted dynamic prices based on their bandwidth requirement and the available capacity. Results: We characterize the structure of the optimal pricing policy as a function of the system state and of the input parameters. Because it is impossible to solve this problem for practically large state spaces, we propose a heuristic dynamic pricing policy that performs very well, particularly when the ratio of capacity to demand rate is low. Managerial implications: We demonstrate the value of using a dynamic heuristic pricing policy compared with the myopic and optimal static policies. The previous literature has studied similar systems with fixed capacity and has characterized conditions under which myopic policies perform well. In contrast, our setting has dynamic (stochastic) capacity, and we find that identifying good state-dependent heuristic pricing policies is of greater importance. Our heuristic policy is computationally more tractable and easier to implement than the optimal dynamic and static pricing policies. It also provides a significant performance improvement relative to the myopic and optimal static policies when capacity is scarce, a condition that holds for the practical setting that motivated this research.

Petri net based multi-robot task coordination from temporal logic specifications

We propose a methodology for enforcing a set of coordination rules onto a multi-robot system, based on the use of Petri nets to model the team of robots, safe linear temporal logic to specify a set of coordination rules to be enforced, and supervisory control theory to synthesise a supervisor that enforces the coordination rules. We introduce a composition algorithm that allows us to build a Petri net that represents the largest restriction of the team behaviour that still satisfies the specification. Such a Petri net can be interpreted as a candidate for a supervisor, for which one needs to verify admissibility. We present a general verification procedure for this problem. We also present a syntactic restriction to safe linear temporal logic that guarantees admissibility of the composition a priori. We finish by providing an illustrative example, where we show how the use of temporal logic allows the designer to write the specifications intuitively, and the use of Petri nets allows us to tackle the large state spaces and high concurrency associated with multi-robot systems.

Multi-period orienteering with uncertain adoption likelihood and waiting at customers

We introduce a multi-period orienteering problem in which a salesperson visits customers to provide information on product and service offerings that may lead to eventual adoptions at the end of the problem horizon. Upon arriving at a customer, the salesperson may experience uncertain wait time that delays or prevents the salesperson from meeting with the customer within the customer’s time window. A customer’s adoption probability evolves stochastically from period to period depending on whether the salesperson meets with the customer. To maximize the expected sales accrued from product adoptions at the end of the horizon, the salesperson must decide in which period(s) to visit the customers and what sequence to visit the customers within each period. We formally model this problem as a Markov decision process. To overcome the computational challenges induced by the large state and action space, we propose a two-stage heuristic approach to facilitate decision making. In the first stage, we solve an assignment problem to determine which customers to visit in the current period. In the second stage, we solve a routing problem to determine the sequence to visit the selected customers. In the computational experiments, we demonstrate the effectiveness of our heuristic methods.

Unifying theories of reactive design contracts

Design-by-contract is an important technique for model-based design in which a composite system is specified by a collection of contracts that specify the behavioural assumptions and guarantees of each component. In this paper, we describe a unifying theory for reactive design contracts that provides the basis for modelling and verification of reactive systems. We provide a language for expression and composition of contracts that is supported by a rich calculational theory. In contrast with other semantic models in the literature, our theory of contracts allows us to specify both the evolution of state variables and the permissible interactions with the environment. Moreover, our model of interaction is abstract, and supports, for instance, discrete time, continuous time, and hybrid computational models. Being based in Unifying Theories of Programming (UTP), our theory can be composed with further computational theories to support semantics for multi-paradigm languages. Practical reasoning support is provided via our proof framework, Isabelle/UTP, including a proof tactic that reduces a conjecture about a reactive program to three predicates, symbolically characterising its assumptions and guarantees about intermediate and final observations. This allows us to verify programs with a large or infinite state space. Our work advances the state-of-the-art in semantics for reactive languages, description of their contractual specifications, and compositional verification.

On Sampling Time Maximization in Wireless Powered Internet of Things

Sensing devices operating in the upcoming Internet of Things (IoT) are likely to rely on the radio frequency (RF) transmissions of a hybrid access point (HAP) for energy. The HAP is also responsible for setting the sampling or monitoring time of these devices according to their harvested energy. This task, however, is made challenging when the HAP has imprecise knowledge of the channel gains to each device. Consequently, the HAP does not know exactly the amount of energy harvested by each device. As a result, the HAP may program a sensing device with an incorrect sampling time. To address this problem, we employ stochastic programming, and use it to determine the time used for charging, and also the sampling time of each device. Its objective is to maximize the minimum sampling time of devices. The formulated stochastic program, however, requires a model or the probability distribution of channel gains. To this end, we propose a reinforcement learning (RL) approach to solve the same problem. In addition, as the state-space contains continuous quantities, we use linear function approximation and a set of novel features to represent the large state-space. Our experiment results show that the RL approach is able to achieve 93% of the minimum sampling time computed by the stochastic program.

Sex-related differences in intrinsic brain dynamism and their neurocognitive correlates

The application of dynamic or time-varying connectivity techniques to neuroimaging data represents a new and complementary method to traditional static (time-averaged) methods, capturing additional patterns of variation in human brain function. Dynamic connectivity and related measures of brain dynamism have been detailed in neurotypical brain function, during human development and across neuropsychiatric disorders, and linked to cognitive control and executive function abilities. Despite this large and growing body of work, little is known about whether sex-related differences are present in dynamic connectivity and brain dynamism, a question pertinent to our understanding of brain function in both health and disease, given the sex bias observed in the prevalence of neuropsychiatric disorders, and well-demonstrated sex-related differences in the performance of certain neurocognitive tasks. We present the first analyses of sex-related effects in dynamic connectivity and brain dynamism referenced to neurocognitive function, in a large sample of sex-, age- and motion-matched subjects in 24- and 51-network whole brain functional parcellations. We demonstrate that sexual dimorphism is present in human dynamic connectivity and in multiple high-order measures of brain dynamism, as well as validating prior work that sex-related differences exist in static intrinsic connectivity. We also provide the first evidence suggesting a link between differential neurocognitive performance by males and females and brain functional dynamics. Reduced dynamism in females, who spend more time in certain brain states and switch states less frequently, may provide a ‘stickier’ functional substrate associated with slower response inhibition, whereas males exhibit greater dynamic fluidity, change between certain states more often and range over a larger state space, achieving superior performance in mental rotation, which demands an iterative visualization and problem-solving approach. We conclude that sex is an important variable to consider in functional MRI experiments and the analysis of dynamic connectivity and brain dynamism.

Modelling and simulation of repairable mechanical systems reliability and availability

Markov approach is applicable for reliability and availability modelling when time to failure and repair follow an exponential distribution. Since failure time of mechanical components follows Weibull distribution, Markov approach cannot be employed to these systems. In present work, Semi-Markov model, which is appropriate for repairable mechanical systems, is considered. In structural dependency, once a unit of a repairable system is failed due to one or more of its constituent components, the entire unit is taken for repair. Therefore, the repairs are considered at the unit level. This feature of structural dependency in the proposed approach addresses the problem of larger state space. The states at the unit level are derived from the component states to develop the system model. The solutions for reliability and availability are obtained using the Monte Carlo simulation. The suggested approach is realistic than the existing software’s such as Rapid Algorithmic Prototyping Tool for Ordered Reasoning, Blocksim, etc., as failures and repairs are considered at different hierarchical levels. The recommended approach is illustrated for a centrifugal pumping system.

Search, Abstractions and Learning in Real-Time Strategy Games

Real-Time Strategy Games’ large state and action spaces pose a significant hurdle to traditional AI techniques. We propose decomposing the game into sub-problems and integrating the partial solutions into action scripts that can be used as abstract actions by a search or machine learning algorithm. The resulting high level algorithm produces sound strategic choices, and can then be combined with a low-level search algorithm to refine tactical choices. We show strong results in SparCraft, Starcraft: Brood War and $$\mu $$RTS against state-of-the-art agents. We expect advances in RTS AI can be used in commercial videogames for playtesting and game balancing, while also having possible real-world applications.

Applying unweighted least‐squares based techniques to stochastic dynamic programming: theory and application

Big data and the curse of dimensionality are common vocabularies that researchers in different communities have recently been dealing with, e.g. dynamic programming (DP) in automatic control system society. A novel unweighted sampled based least square projection approach is proposed in this study to address the issue of the large state space in the DP optimisation problem. The method, in particular, takes into account both contraction mapping and monotonicity properties of the DP algorithm for value function approximation. Specifically, the batch of samples are gathered by uniform probability distribution at first, and an unweighted LS sub-problem in the subspace is solved. As the case study, a new Markov decision process model associated with a resource allocation problem is considered to illustrate the technique and evaluate its effectiveness. It is noted that the approach can be employed for different applications as well. Moreover, a MATLAB based software is developed to implement and examine different parts of the proposed method. Simulation examples are considered to support the results of the approach via developed software. The idea makes a connection between the recent advances in big data analysis and approximate DP as well.

Energy Optimization Dispatching of Microgrid in The Stand-alone Mode

A microgrid system in the stand-alone mode with photovoltaic, wind turbine, microturbine, fuel cell and energy storage unit is studied. Mathematical models of different distributed power supply and energy storage devices are established. The load demand of the system, the price of purchasing and selling electricity when interacting with the power grid and other constraints are considered. An economic model for optimizing stand-alone operation cost is established. According to the characteristics of the model, an improved genetic algorithm is selected to optimize the nonlinear system. The algorithm has better stability in solving the optimization problem of large state space dimension. The case study show that the proposed economic model is reasonable and the algorithm is effective.

Solving the Rubik’s cube with deep reinforcement learning and search

The Rubik’s cube is a prototypical combinatorial puzzle that has a large state space with a single goal state. The goal state is unlikely to be accessed using sequences of randomly generated moves, posing unique challenges for machine learning. We solve the Rubik’s cube with DeepCubeA, a deep reinforcement learning approach that learns how to solve increasingly difficult states in reverse from the goal state without any specific domain knowledge. DeepCubeA solves 100% of all test configurations, finding a shortest path to the goal state 60.3% of the time. DeepCubeA generalizes to other combinatorial puzzles and is able to solve the 15 puzzle, 24 puzzle, 35 puzzle, 48 puzzle, Lights Out and Sokoban, finding a shortest path in the majority of verifiable cases. For some combinatorial puzzles, solutions can be verified to be optimal, for others, the state space is too large to be certain that a solution is optimal. A new deep learning based search heuristic performs well on the iconic Rubik’s cube and can also generalize to puzzles in which optimal solvers are intractable.

Hierarchical reinforcement learning via dynamic subspace search for multi-agent planning

We consider scenarios where a swarm of unmanned vehicles (UxVs) seek to satisfy a number of diverse, spatially distributed objectives. The UxVs strive to determine an efficient plan to service the objectives while operating in a coordinated fashion. We focus on developing autonomous high-level planning, where low-level controls are leveraged from previous work in distributed motion, target tracking, localization, and communication. We rely on the use of state and action abstractions in a Markov decision processes framework to introduce a hierarchical algorithm, Dynamic Domain Reduction for Multi-Agent Planning, that enables multi-agent planning for large multi-objective environments. Our analysis establishes the correctness of our search procedure within specific subsets of the environments, termed ‘sub-environment’ and characterizes the algorithm performance with respect to the optimal trajectories in single-agent and sequential multi-agent deployment scenarios using tools from submodularity. Simulated results show significant improvement over using a standard Monte Carlo tree search in an environment with large state and action spaces.

Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning

In this letter, we focus on finding practical resolution methods for Markov decision processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: first, having to deal with huge state spaces; and second, designing a method that is robust enough to dead ends. These complications restrict or make more difficult the application of methods, such as value iteration, policy iteration, or labeled real-time dynamic programming (LRTDP). We see in determinization and heuristic search a way to successfully work around these problems. In addition, we believe that many practical use cases offer the opportunity to identify hierarchies of subtasks and solve smaller, simplified problems. We propose a decision-making unit that operates in a probabilistic planning setting through stochastic shortest path problems, which generalize the most common types of MDPs. Our decision-making unit combines: first, automatic hierarchical organization of subtasks; and second, on-line resolution via determinization. We argue that several applications of planning benefit from these two strategies. We exemplify our approach with a robotized disassembly application. The disassembly problem is modeled in probabilistic planning definition language, and serves to define our experiments. Our results show many advantages of our method over LRTDP, such as a better capability to handle problems with large state spaces and state definitions that change when new fluents are discovered.

Moving Forward: A Simulation-Based Approach for Solving Dynamic Resource Management Problems

Standard dynamic resource optimization approaches, such as value function iteration, are challenged by problems involving complex uncertainty and a large state space. We extend a solution technique to address these limitations called approximate dynamic programming (ADP). ADP recently emerged in the macroeconomics literature and is novel to bioeconomics. We demonstrate ADP in solving a simple fishery management model under uncertainty to show: the mechanics of ADP in simplest form; the accuracy of ADP; the value of a nonparametric extension; and readily adaptable, non-specialized code. We then demonstrate ADP’s capacity to handle rich bioeconomic problems by solving the fishery management problem subject to four autocorrelated shock processes (governing economic returns and biological dynamics) which entails four sources of stochasticity and five continuous state variables. We find that accounting for multiple autocorrelation has important impacts on harvest policy and generates gains that depend crucially on the structure of harvest cost.

Verifying SystemC Using Intermediate Verification Language and Stateful Symbolic Simulation

Formal verification of high-level SystemC designs is an important and challenging problem. One has to deal with the full complexity of C++ to extract a suitable formal model (front-end problem) and then, with large cyclic state spaces defined by symbolic inputs and concurrent processes. This paper describes a scalable and efficient stateful symbolic simulation approach for SystemC that combines state subsumption reduction (SSR) with partial order reduction (POR) and symbolic execution (SymEx) under the SystemC simulation semantics. While the SymEx+POR combination provides basic capabilities to efficiently explore the state space, SSR prevents revisiting symbolic states and therefore makes the verification complete. The approach has been implemented on top of an intermediate verification language for SystemC to address the front-end problem. The scalability and efficiency of the implemented verifier is demonstrated using an extensive set of experiments.

Double Deep $Q$ -Learning-Based Distributed Operation of Battery Energy Storage System Considering Uncertainties

$Q$ -learning-based operation strategies are being recently applied for optimal operation of energy storage systems, where, a $Q$ -table is used to store $Q$ -values for all possible state-action pairs. However, $Q$ -learning faces challenges when it comes to large state space problems, i.e., continuous state space problems or problems with environment uncertainties. In order to address the limitations of $Q$ -learning, this paper proposes a distributed operation strategy using double deep $Q$ -learning method. It is applied to managing the operation of a community battery energy storage system (CBESS) in a microgrid system. In contrast to $Q$ -learning, the proposed operation strategy is capable of dealing with uncertainties in the system in both grid-connected and islanded modes. This is due to the utilization of a deep neural network as a function approximator to estimate the $Q$ -values. Moreover, the proposed method can mitigate the overestimation that is the major drawback of the standard deep $Q$ -learning. The proposed method trains the model faster by decoupling the selection and evaluation processes. Finally, the performance of the proposed double deep $Q$ -learning-based operation method is evaluated by comparing its results with a centralized approach-based operation.

Hierarchical Sarsa Learning Based Route Guidance Algorithm

In modern society, route guidance problems can be found everywhere. Reinforcement learning models can be normally used to solve such kind of problems; particularly, Sarsa Learning is suitable for tackling with dynamic route guidance problem. But how to solve the large state space of digital road network is a challenge for Sarsa Learning, which is very common due to the large scale of modern road network. In this study, the hierarchical Sarsa learning based route guidance algorithm (HSLRG) is proposed to guide vehicles in the large scale road network, in which, by decomposing the route guidance task, the state space of route guidance system can be reduced. In this method, Multilevel Network method is introduced, and Differential Evolution based clustering method is adopted to optimize the multilevel road network structure. The proposed algorithm was simulated with several different scale road networks; the experiment results show that, in the large scale road networks, the proposed method can greatly enhance the efficiency of the dynamic route guidance system.

Learning moment closure in reaction-diffusion systems with spatial dynamic Boltzmann distributions.

Many physical systems are described by probability distributions that evolve in both time and space. Modeling these systems is often challenging due to their large state space and analytically intractable or computationally expensive dynamics. To address these problems, we study a machine-learning approach to model reduction based on the Boltzmann machine. Given the form of the reduced model Boltzmann distribution, we introduce an autonomous differential equation system for the interactions appearing in the energy function. The reduced model can treat systems in continuous space (described by continuous random variables), for which we formulate a variational learning problem using the adjoint method to determine the right-hand sides of the differential equations. This approach can be used to enforce a reduced physical model by a suitable parametrization of the differential equations. The parametrization we employ uses the basis functions from finite-element methods, which can be used to model any physical system. One application domain for such physics-informed learning algorithms is to modeling reaction-diffusion systems. We study a lattice version of the Rössler chaotic oscillator, which illustrates the accuracy of the moment closure approximation made by the method and its dimensionality reduction power.

Reinforcement learning-based cell selection in sparse mobile crowdsensing

Sparse Mobile Crowdsensing (MCS) is a novel MCS paradigm which allows us to use the mobile devices to collect sensing data from only a small subset of cells (sub-areas) in the target sensing area while intelligently inferring the data of other cells with quality guarantee. Since selecting sensed data from different cell sets will probably lead to diverse levels of inference data quality, cell selection (i.e., choosing which cells in the target area to collect sensed data from participants) is a critical issue that will impact the total amount of data that requires to be collected (i.e., data collection costs) for ensuring a certain level of data quality. To address this issue, this paper proposes the reinforcement learning-based cell selection algorithm for Sparse MCS. First, we model the key concepts in reinforcement learning including state, action, and reward, and then propose a Q-learning based cell selection algorithm. To deal with the large state space, we employ the deep Q-network to learn the Q-function that can help decide which cell is a better choice under a certain state during cell selection. Then, we modify the Q-network to a deep recurrent Q-network with LSTM to catch the temporal patterns and handle partial observability. Furthermore, we leverage the transfer learning techniques to relieve the dependency on a large amount of training data. Experiments on various real-life sensing datasets verify the effectiveness of our proposed algorithms over the state-of-the-art mechanisms in Sparse MCS by reducing up to 20% of sensed cells with the same data inference quality guarantee.