Monte Carlo Tree Search Algorithm Research Articles

A Monte Carlo tree search-based method for online decision making of generator startup sequence considering hot start

Generator startup is a critical step for the power system restoration following blackouts. Most strategies for the startup sequence of generators are obtained by offline methods, which probably are ineffective in the real situation: some devices unexpectedly fail to be restarted. To deal with the uncertainties, this paper proposed a novel method for online decision-making of the startup sequenceof generators. First, the framework of online decision making is presented. The principles of online decision-making of the generator startup sequence are analyzed. Second, an optimization model for determining the generator startup sequence is developed to maximize the generating capacity. Then, the Monte Carlo tree search algorithm (MCTS) is applied to online decide the generator to be restored in the next step according to the real-time situation. To make it more suitable for system restoration, MCTS is modified: the upper confidence bounds for trees, the default policy, backpropagation, and the final decision making are improved. The restoration paths related to the selected generator are obtained by a graphic theoretic algorithm. Except commonly used constraints on generator startup and the path restoration, this paper considers the hot start of generators to propose a constraint about the robustness of the restoration scheme. Finally, the feasible strategy with the maximum generating capacity is selected online for application. Case studies on the IEEE 39-bus test system and an actual power system in Hebei, China, demonstrated the effectiveness of this method.

Monte Carlo Tree Search in Continuous Spaces Using Voronoi Optimistic Optimization with Regret Bounds

Many important applications, including robotics, data-center management, and process control, require planning action sequences in domains with continuous state and action spaces and discontinuous objective functions. Monte Carlo tree search (MCTS) is an effective strategy for planning in discrete action spaces. We provide a novel MCTS algorithm (voot) for deterministic environments with continuous action spaces, which, in turn, is based on a novel black-box function-optimization algorithm (voo) to efficiently sample actions. The voo algorithm uses Voronoi partitioning to guide sampling, and is particularly efficient in high-dimensional spaces. The voot algorithm has an instance of voo at each node in the tree. We provide regret bounds for both algorithms and demonstrate their empirical effectiveness in several high-dimensional problems including two difficult robotics planning problems.

A novel approach of dynamic base station switching strategy based on Markov decision process for interference alignment in VANETs

In Vehicular Ad-hoc Networks (VANETs), high frequent interaction of safety-related information is required among vehicles and imposes urgent demand on information update. In order to reduce communication delay and improve the capacity of concurrent communication, in this paper we propose that the multi-antenna vehicle could takeover the channel management as dynamic base station to apply Multiple-Input Multiple-Output communication and interference management approach in Vehicle to Vehicle (V2V) communications. Firstly, we construct an Markov decision process (MDP) model for multi-antenna vehicle to estimate whether it is appropriate to be dynamic base station. In addiction, Monte Carlo Tree Search algorithm is introduced to derive MDP policy. Thirdly, the V2V Interference Alignment (V2V-IA) model is constructed for dynamic base station to obtain IA scheme to manage V2V communications and IA in VANETs. To achieve the goal of improving frequency of information update, we propose an optimized problem to minimize total number of time slots, which is required for completing global safety-related information delivery. Simulation results show that the frequency of information update can be improved effectively by the proposed approach and the average improvement could go up to 40%.

Efficient Construction of a Chemical Reaction Network Guided By a Monte Carlo Tree Search

AbstractChemical reaction networks are essential for the complete elucidation of chemical reaction mechanisms. Graph‐theoretic methods combined with quantum calculations are known to be an efficient approach with broad applicability for constructing reaction networks. However, this method entails high computational cost due to quantum calculations on chemically irrelevant intermediates coming from the exploration of a large scale chemical space. To remedy this problem, we propose to apply the Monte Carlo tree search algorithm to those graph‐theoretic methods. We have combined it with ACE‐Reaction, our in‐house graph‐theoretic method, and demonstrated its performance on five organic reactions. The result shows that the computational cost for quantum calculations has been reduced by up to 75 % from that of the original ACE‐Reaction. Furthermore, cost reduction became more efficient for more complex reactions.

Graph-Based Node Finding in Big Complex Contextual Social Graphs

Graph pattern matching is to find the subgraphs matching the given pattern graphs. In complex contextual social networks, considering the constraints of social contexts like the social relationships, the social trust, and the social positions, users are interested in the top-K matches of a specific node (denoted as the designated node) based on a pattern graph, rather than the entire set of graph matching. This inspires the conText-Aware Graph pattern-based top-K designated node matching (TAG-K) problem, which is NP-complete. Targeting this challenging problem, we propose a recurrent neural network- (RNN-) based Monte Carlo Tree Search algorithm (RN-MCTS), which automatically balances exploring new possible matches and extending existing matches. The RNN encodes the subgraph and maps it to a policy which is used to guide the MCTS. The experimental results demonstrate that our proposed algorithm outperforms the state-of-the-art methods in terms of both efficiency and effectiveness.

Optimal production ramp‐up in the smartphone manufacturing industry

AbstractMotivated by challenges in the smartphone manufacturing industry, we develop a dynamic production ramp‐up model that can be applied to economically satisfy nonstationary demand for short‐life‐cycle products by high‐tech companies. Due to shorter life cycles and more rapid evolution of smartphones, production ramp‐up has been increasingly critical to the success of a new smartphone. In the production ramp‐up, the key challenge is to match the increasing capacity to nonstationary demand. The high‐tech smartphone manufacturers are urged to jointly consider the effect of increasing capacity and decreasing demand. We study the production planning problem using a high‐dimensional Markov decision process (MDP) model to characterize the production ramp‐up. To address the curse of dimensionality, we refine Monte Carlo tree search (MCTS) algorithm and theoretically analyze its convergence and computational complexity. In a real case study, we find that the MDP model achieves revenue improvement by stopping producing the existing product earlier than the benchmark policy. In synthetic instances, we validate that the proposed MCTS algorithm saves computation time without loss of solution quality compared with traditional value iteration algorithm. As part of the Lenovo production solution, our MDP model enables high‐tech smartphone manufacturers to better plan the production ramp‐up.

A Monte Carlo tree search for traveling salesman problem with drone

Abstract The use of drones and trucks working collaboratively has gained drastically attentions in recent years. We develop a new Monte Carlo Tree Search algorithm (MCTS) to solve the Traveling Salesman Problem with Drone (TSP-D) arising in the management of parcel last-mile-delivery systems. The approach seeks to find optimal decisions by taking random samples in decision space and building a search tree based on the results. We address several major issues in adapting the tree search concept to solve the TSP-D in this paper. The solution approach, aimed to minimize the total completion time, is tested on various problem instances derived from the well-known TSPLIB benchmark. Experimental results show a very promising performance of the proposed search method where it outperforms the compared heuristic method, providing new best solutions for 23 instances and an average 12% improvement in terms of solution quality.

Adversarial Examples Detection for XSS Attacks Based on Generative Adversarial Networks

Models based on deep learning are prone to misjudging the results when faced with adversarial examples. In this paper, we propose an MCTS-T algorithm for generating adversarial examples of cross-site scripting (XSS) attacks based on Monte Carlo tree search (MCTS) algorithm. The MCTS algorithm enables the generation model to provide a reward value that reflects the probability of generative examples bypassing the detector. To guarantee the antagonism and feasibility of the generative adversarial examples, the bypassing rules are restricted. The experimental results indicate that the missed detection rate of adversarial examples is significantly improved after the MCTS-T generation algorithm. Additionally, we construct a generative adversarial network (GAN) to optimize the detector and improve the detection rate when dealing with adversarial examples. After several epochs of adversarial training, the accuracy of detecting adversarial examples is significantly improved.

Accelerating copolymer inverse design using monte carlo tree search.

There exists a broad class of sequencing problems in soft materials such as proteins and polymers that can be formulated as a heuristic search that involves decision making akin to a computer game. AI gaming algorithms such as Monte Carlo tree search (MCTS) gained prominence after their exemplary performance in the computer Go game and are decision trees aimed at identifying the path (moves) that should be taken by the policy to reach the final winning or optimal solution. Major challenges in inverse sequencing problems are that the materials search space is extremely vast and property evaluation for each sequence is computationally demanding. Reaching an optimal solution by minimizing the total number of evaluations in a given design cycle is therefore highly desirable. We demonstrate that one can adopt this approach for solving the sequencing problem by developing and growing a decision tree, where each node in the tree is a candidate sequence whose fitness is directly evaluated by molecular simulations. We interface MCTS with MD simulations and use a representative example of designing a copolymer compatibilizer, where the goal is to identify sequence specific copolymers that lead to zero interfacial energy between two immiscible homopolymers. We apply the MCTS algorithm to polymer chain lengths varying from 10-mer to 30-mer, wherein the overall search space varies from 210 (1024) to 230 (∼1 billion). In each case, we identify a target sequence that leads to zero interfacial energy within a few hundred evaluations demonstrating the scalability and efficiency of MCTS in exploring practical materials design problems with exceedingly vast chemical/material search space. Our MCTS-MD framework can be easily extended to several other polymer and protein inverse design problems, in particular, for cases where sequence-property data is either unavailable and/or is resource intensive.

Analysis of Statistical Forward Planning Methods in Pommerman

Pommerman is a complex multi-player and partially observable game where agents try to be the last standing to win. This game poses very interesting challenges to AI, such as collaboration, learning and planning. In this paper, we compare two Statistical Forward Planning algorithms, Monte Carlo Tree Search (MCTS) and Rolling Horizon Evolutionary Algorithm (RHEA) in Pommerman. We provide insights on how the agents actually play the game, inspecting their behaviours to explain their performance. Results show that MCTS outperforms RHEA in several game settings, but leaving room for multiple avenues of future work: tuning these methods, improving opponent modelling, identifying trap moves and introducing of assumptions for partial observability settings.

Analysis and Comparison of Monte Carlo Tree Search versus ACO Algorithms in Distribution of Resources and Environment

Swarm intelligence (CI) has been widely studied in the past decades. The most famous CI algorithm is ant colony algorithm (ACO), which is used to solve complex path search problems. AlphaZero program, through self-reinforcement learning of Monte Carlo tree algorithm from scratch, has achieved transcendental results in go, chess and general chess. By analyzing and comparing the Monte Carlo Tree Search (MCTS) and ACO algorithms on Distribution of resources and environment, the reasons for AlphaZero’s success are revealed. It is not only because of deep neural networks and intensive learning, but also because the algorithm is essentially an evolutionary algorithm of CI emergence.

Analysis of Hannan consistent selection for Monte Carlo tree search in simultaneous move games

Hannan consistency, or no external regret, is a key concept for learning in games. An action selection algorithm is Hannan consistent (HC) if its performance is eventually as good as selecting the best fixed action in hindsight. If both players in a zero-sum normal form game use a Hannan consistent algorithm, their average behavior converges to a Nash equilibrium of the game. A similar result is known about extensive form games, but the played strategies need to be Hannan consistent with respect to the counterfactual values, which are often difficult to obtain. We study zero-sum extensive form games with simultaneous moves, but otherwise perfect information. These games generalize normal form games and they are a special case of extensive form games. We study whether applying HC algorithms in each decision point of these games directly to the observed payoffs leads to convergence to a Nash equilibrium. This learning process corresponds to a class of Monte Carlo Tree Search algorithms, which are popular for playing simultaneous-move games but do not have any known performance guarantees. We show that using HC algorithms directly on the observed payoffs is not sufficient to guarantee the convergence. With an additional averaging over joint actions, the convergence is guaranteed, but empirically slower. We further define an additional property of HC algorithms, which is sufficient to guarantee the convergence without the averaging and we empirically show that commonly used HC algorithms have this property.

A game-based approximate verification of deep neural networks with provable guarantees

Despite the improved accuracy of deep neural networks, the discovery of adversarial examples has raised serious safety concerns. In this paper, we study two variants of pointwise robustness, the maximum safe radius problem, which for a given input sample computes the minimum distance to an adversarial example, and the feature robustness problem, which aims to quantify the robustness of individual features to adversarial perturbations. We demonstrate that, under the assumption of Lipschitz continuity, both problems can be approximated using finite optimisation by discretising the input space, and the approximation has provable guarantees, i.e., the error is bounded. We then show that the resulting optimisation problems can be reduced to the solution of two-player turn-based games, where the first player selects features and the second perturbs the image within the feature. While the second player aims to minimise the distance to an adversarial example, depending on the optimisation objective the first player can be cooperative or competitive. We employ an anytime approach to solve the games, in the sense of approximating the value of a game by monotonically improving its upper and lower bounds. The Monte Carlo tree search algorithm is applied to compute upper bounds for both games, and the Admissible A⁎ and the Alpha-Beta Pruning algorithms are, respectively, used to compute lower bounds for the maximum safety radius and feature robustness games. When working on the upper bound of the maximum safe radius problem, our tool demonstrates competitive performance against existing adversarial example crafting algorithms. Furthermore, we show how our framework can be deployed to evaluate pointwise robustness of neural networks in safety-critical applications such as traffic sign recognition in self-driving cars.

Tackling Sparse Rewards in Real-Time Games with Statistical Forward Planning Methods

One of the issues general AI game players are required to deal with is the different reward systems in the variety of games they are expected to be able to play at a high level. Some games may present plentiful rewards which the agents can use to guide their search for the best solution, whereas others feature sparse reward landscapes that provide little information to the agents. The work presented in this paper focuses on the latter case, which most agents struggle with. Thus, modifications are proposed for two algorithms, Monte Carlo Tree Search and Rolling Horizon Evolutionary Algorithms, aiming at improving performance in this type of games while maintaining overall win rate across those where rewards are plentiful. Results show that longer rollouts and individual lengths, either fixed or responsive to changes in fitness landscape features, lead to a boost of performance in the games during testing without being detrimental to non-sparse reward scenarios.

Fast preliminary design of low-thrust trajectories for multi-asteroid exploration

Multiple-asteroid exploration with low-thrust propulsion requires the design of transfer trajectory and the selection of the visiting sequence, as well as the estimation of the propellant budget. To this end, this paper presents a method to generate transfer trajectories and visiting sequences rapidly using finite Fourier series (FFS) and Monte-Carlo Tree Search (MCTS). The FFS method can generate the transfer trajectory rapidly, which can provide suitable initial approximations, leading to more accurate trajectory optimizations. This study adopts the MCTS algorithm for asteroid sequence selection in the multi-asteroid exploration. By comparing with the traversal algorithm, the greedy algorithm and the tree search algorithm with the trimming strategy, the numerical results show that the MCTS can be used to obtain a quasi-optimal sequence with higher probability and less computation time. Consequently, a method combining FFS and MCTS can rapidly acquire the quasi-optimal visiting sequences with a large probability and the suitable initial trajectory for a multi-asteroid exploration mission. This is very important for the rapid feasibility assessment of hundreds of flight scenarios at the preliminary mission design stage.

Optimizing treatment combination for lymphoma using an optimization heuristic

Background. The standard treatment for high-grade non-Hodgkin lymphoma involves the combination of chemotherapy and immunotherapy. We characterize in-silico the optimal combination protocol that maximizes the overall survival probability. We rely on a pharmacokinetics/pharmacodynamics (PK/PD) model that describes the joint evolution of tumor and effector cells, as well as the effects of both chemotherapy and immunotherapy. The toxicity is taken into account through ad-hoc constraints. We develop an optimization algorithm that belongs to the class of Monte-Carlo tree search algorithms. Our simulations rely on an in-silico population of heterogeneous patients differing with respect to their PK/PD parameters. The optimization objective consists in characterizing the combination protocol that maximizes the overall survival probability of the patient population under consideration.Results. We compare using in-silico experiments our results to standard protocols and observe a gain in overall survival probabilities that vary from 4 to 9 percentage points. The gains increase with the complexity of the potential protocol. Gains are larger in presence of a higher number of injections or of an actual combination with immunotherapy.Conclusions. In in-silico experiments, optimal protocols achieve significant gains over standard protocols when considering overall survival probabilities. Our optimization algorithm enables us to efficiently tackle this numerical problem with a large dimensionality. The in-vivo implications of our in-silico results remain to be explored.

Utilization Management and Pricing of Parking Facilities Under Uncertain Demand and User Decisions

Excessive search for parking spots in congested areas contributes to additional travel delays and negative socio-economic impacts. While managing parking utilization to address the agency’s objectives, it is often very beneficial to reflect the diversity of users’ behaviors and their travel choices. This paper develops a stochastic dynamic parking management model, under competitive user-agency perceptions and uncertain user demand and parking occupancy, to simultaneously minimize the total travelers’ costs and maximize the parking agency’s revenue. The problem is formulated into a dynamic programming model and solved using a stochastic look-ahead technique based on the Monte Carlo tree search algorithm to determine optimal actions on parking price assignment and spot utilization over time. The numerical experiments on a hypothetical and empirical case study are conducted to show the performance of the proposed algorithm and to draw managerial insights. The results are compared with those of benchmark algorithms, which indicate that the proposed methodology can determine near-optimal solutions efficiently.

Some improvements in Monte Carlo tree search algorithms for sudden death games

The average reward problem in the traditional Monte-Carlo tree search algorithms troubles the sudden death games for a long time, because of the average reward criterion will reduce the probability of a deterministic result. But this does not bother the non-sudden death games, such as Go, which can focus only on higher win rates rather than higher scores. In this work, we propose the miniMax-Monte-Carlo tree search with depth rewards to Outer-Open Gomoku (a variant of Gomoku) to discover a forced win/lose without any human knowledge, evaluation function, or pre-training. And it can solve not only the average reward problem but also the inaccurate win-rate problem in deep playout simulation in sudden death games. Finally, we propose a new integrated framework named BBQ (Big, Best, Quick win) MCTS for improving the performance of traditional MCTS.

A Variated Monte Carlo Tree Search Algorithm for Automatic Performance Tuning to Achieve Load Scalability in InnoDB Storage Engines

A Variated Monte Carlo Tree Search Algorithm for Automatic Performance Tuning to Achieve Load Scalability in InnoDB Storage Engines

Frequency Selection Approach for Energy Aware Cloud Database

A lot of cloud systems are adopted in industry and academia to face the explosion of the data volume and the arrival of the big data era. Meanwhile, energy efficiency and energy saving become major concerns for data centers where massive cloud systems are deployed. However, energy waste is quite common due to resource provisioning. In this paper, using dynamic voltage and frequency scaling (DVFS), a frequency selection approach is introduced to improve the energy efficiency of the cloud systems in terms of resource provisioning. In the approach, two algorithms, genetic algorithm (GA) and Monte Carlo tree search algorithm (MCTS), are proposed. A cloud database system is taken as an example to evaluate the approach. The results of the experiments show that both algorithms have its advantages. The algorithms have great scalability, in which the GA can be applied to thousands of nodes and the MCTS can be applied to hundreds of nodes. Both algorithms have high accuracy compared with optimal solutions (up to 99.9% and 99.6% for GA and MCTS, respectively). According to an optimality bound analysis, 26% of energy can be saved at most using our frequency selection approach.