A Survey of Monte Carlo Tree Search Methods

This paper is devoted to the scientific problem of improvements of the Monte Carlo Tree Search (MCTS) method. The object of research is the process of performing a tree search using the MCTS. The subject of research is the MCST improvement technique with control of the search tree shape by usage of the previously proposed be the authors DWC (Depth/Width Criterion) and WDC (Width/Depth Criterion) criteria. This technique was named Monte Carlo Tree Search with Tree Shape Control (MCTS-TSC). The research methods are based on the theory of data structures and analysis methods. The aim of the study is to conduct extended study of the previously proposed MCTS-TSC technique for improvement of the MCTS method. In particular, the aim is to approve that the DWC and WDC tree shape control criteria ensure the better move selection and increasing player strength compared to the standard Monte Carlo Tree Search with Upper Confidence bounds applied to Trees (MCTS-UCT) technique. To achieve the aim, the following tasks were set: to conduct a set of experiments according to the developed approbation methodology to approve that the WDC criterion of the MCTS-TSC technique is able to improve the MCTS method; to conduct a set of experiments according to the developed approbation methodology to approve that the DWC criterion of the MCTS-TSC technique is able to improve the MCTS method. Both WDC and DWC criteria of the MCTS-TSC technique were tested on a series of games of Connect Four between a player, which used the MCTS-TSC technique, and a player which used the MCTS-UCT technique. Different parameters for tuning the formulas of the WDC and DWC criteria of the MCTS-TSC technique were used in the experiments. The paper describes the methodology of the approbation of the MCTS-TSC technique with usage of the WDC and DWC criteria compared to the MCTS-UCT technique and conducts comparative analysis of the results of the experiments. The MCTS-TSC player won from 30% to 70% more games than the MCTS-UCT player for some search tree shapes, when WDC criterion was used, and from 19% to 52% more games, when DWC criterion was used. So, ability of the proposedMCTS-TSC technique to improve the MCTS method was approved for both criteria, WDC and DWC. Key words: depth-width type criteria, Monte Carlo tree search method, MCTS, MCTS-UCT, MCTS-TSC, search tree shape control.

Greedy heuristics may be attuned by looking ahead for each possible choice, in an approach called the rollout or Pilot method. These methods may be seen as meta-heuristics that can enhance (any) heuristic solution, by repetitively modifying a master solution: similarly to what is done in game tree search, better choices are identified using lookahead, based on solutions obtained by repeatedly using a greedy heuristic. This paper first illustrates how the Pilot method improves upon some simple well known dispatch heuristics for the job-shop scheduling problem. The Pilot method is then shown to be a special case of the more recent Monte Carlo Tree Search (MCTS) methods: Unlike the Pilot method, MCTS methods use random completion of partial solutions to identify promising branches of the tree. The Pilot method and a simple version of MCTS, using the ε-greedy exploration paradigms, are then compared within the same framework, consisting of 300 scheduling problems of varying sizes with fixed-budget of rollouts. Results demonstrate that MCTS reaches better or same results as the Pilot methods in this context.

In this work, we are dedicated to multi-target active object tracking (AOT), where the goal is to achieve continuous tracking of targets through real-time control of camera. This form of active camera control can be applied to unmanned aerial vehicles (UAV), intelligent robots, and sports events. Our work is conducted in an environment featuring multiple cameras and targets, where our goal is to maximize target coverage. Contrasting with previous research, our work introduces additional degrees of freedom for the cameras, allowing them not only to rotate but also to move along boundary lines. In addition, we model the motion of target to predict the future position of the target in environment. With target’s future position, we use Monte Carlo Tree Search (MCTS) method to find the optimal action of camera. Since the action space is large, we propose to leverage the action selection from multi-agent reinforcement learning (MARL) network to prune the search tree of Monte Carlo Tree Search method, so as to find the optimal action more efficiently. We establish a multi-target 2D environment to simulate several sports games, and experimental results demonstrate that our method can effectively improve the target coverage. The code is available at: http://github.com/HopeChanger/ActiveObjectTracking .

This article explores how to improve operational performance in maritime ports by managing the flow of goods effectively. This study proposes an innovative approach based on Reinforcement Learning (RL), specifically the Monte Carlo Tree Search (MCTS) method, to address the restricted container relocation problem (RCRP). This method aims to determine an optimal sequence for container retrieval based on their respective priorities, in order to minimize the number of necessary relocations. By employing precise actions and a defined reward function, MCTS is guided towards the best possible solution. The efficiency and relevance of this method are demonstrated through various solved scenarios and compared to a literature-based approach using genetic algorithms. The results show that the MCTS approach is effective in addressing the complex challenges of goods flow management in maritime ports.

In this non-comparative study paper, local regression modeling is added to Monte Carlo Tree Search (MCTS). This local regression modeling is used to replace the random action selection in the simulation part of MCTS. The modified MCTS method and the regular MCTS method were tested against each other on the tic-tac-toe and the connect four games. The results of the experiment indicated that local regression helped the modified MCTS to outperform the regular MCTS on moderately memory demanding games, such as connect four. The performances of the two approaches appear to equal each other on low branching factor and less memory demanding games similar to tic-tac-toe.

Task scheduling is critical for improving system performance in the distributed heterogeneous computing environment. The Directed Acyclic Graph (DAG) tasks scheduling problem is NP-complete and it is hard to find an optimal schedule. Due to its key importance, the DAG tasks scheduling problem has been extensively studied in the literature. However, many previously proposed traditional heuristic algorithms are usually based on greedy methods and also lack the consideration of scheduling tasks between trusted and untrusted entities, which makes the problem more complicated, but there still exists a large optimization space to be explored. In this paper, we propose a trust-aware adaptive DAG tasks scheduling algorithm using the reinforcement learning and Monte Carlo Tree Search (MCTS) methods. The scheduling problem is defined using the reinforcement learning model. Efficient scheduling state space, action space and reward function are designed to train the policy gradient-based REINFORCE agent. The MCTS method is proposed to determine actual scheduling policies when DAG tasks are simultaneously executed in trusted and untrusted entities. Leveraging the algorithm’s capability of exploring long term reward, the proposed algorithm could achieve good scheduling policies while guaranteeing trusted tasks scheduled within trusted entities. Experimental results showed the effectiveness of the proposed algorithm compared with the classic HEFT/CPOP algorithms.

A 3D Monte Carlo tree search method for railway alignment optimization

As a critical technology in network virtualization, virtual network embedding (VNE) focuses on how to allocate physical resources to virtual network requests efficiently. Because the VNE problem is NP-hard, most of the existing approaches are heuristic-based algorithms that tend to converge to a local optimal solution and have a low performance. In this paper, we propose an algorithm that combines the basic Monte Carlo tree search (MCTS) method with node importance to apply domain-specific knowledge. For a virtual network request, we first model the embedding process as a finite Markov decision process (MDP), where each virtual node is embedded in one state in the order of node importance that we design. The shortest-path algorithm is then applied to embed links in the terminal state and return the cost as a part of the reward. Due to the reward delay mechanism of the MDP, the result of link mapping can affect the action selected in the previous node mapping stage, coordinating the two embedding stages. With node importance, domain-specific knowledge can be used in the Expansion and Simulation stages of MCTS to speed up the search and estimate the simulation value more accurately. The experimental results show that, compared with the existing classic algorithms, our proposed algorithm can improve the performance of VNE in terms of the average physical node utilization ratio, acceptance ratio, and long-term revenue to cost ratio.

Monte Carlo tree search (MCTS) methods have had recent success in games, planning, and optimization. MCTS uses results from rollouts to guide search; a rollout is a path that descends the tree with a randomized decision at each ply until reaching a leaf. MCTS results can be strongly influenced by the choice of appropriate policy to bias the rollouts. Most previous work on MCTS uses static uniform random or domain-specific policies. We describe a new MCTS method that dynamically adapts the rollout policy during search, in deterministic optimization problems. Our starting point is Cazenave's original Nested Monte Carlo Search (NMCS), but rather than navigating the tree directly we instead use gradient ascent on the rollout policy at each level of the nested search. We benchmark this new Nested Rollout Policy Adaptation (NRPA) algorithm and examine its behavior. Our test problems are instances of Crossword Puzzle Construction and Morpion Solitaire. Over moderate time scales NRPA can substantially improve search efficiency compared to NMCS, and over longer time scales NRPA improves upon all previous published solutions for the test problems. Results include a new Morpion Solitaire solution that improves upon the previous human-generated record that had stood for over 30 years.

Existing point cloud classifiers concern on handling irregular data structures to discover a global and discriminative configuration of local geometries. These classification methods design a number of effective permutation-invariant feature encoding kernels, but still suffer from the intrinsic challenge of large geometric feature variations caused by inconsistent point distributions along object surface. In this paper, point cloud classification can be addressed via deep graph representation learning on aggregating multiple convolutional feature kernels (namely, a poly convolutional operation) anchored on each point with its local neighbours. Inspired by recent success of neural architecture search, we introduce a novel concept of poly-convolutional architecture search (PolyConv search in short) to model local geometric patterns in a more flexible manner. To this end, the Monte Carlo Tree Search (MCTS) method is adopted, which can be formulated into a Markov Decision Process problem to cast decisions for dependently selecting layer-wise aggregation kernels. Experiments on the popular ModelNet40 benchmark have verified that superior performance can be achieved by constructing networks via the MCTS method, with aggregation kernels in our PolyConv search space.

: In this project examine applications of Monte Carlo tree search, an artificial intelligence algorithm, in military simulation environments and assignment and scheduling problems with the goal of enhancing mission command analysis capabilities. We provide a review of recent literature on Monte Carlo tree search methods and then develop two algorithms that adapt the Monte Carlo tree search algorithm, traditionally applied to deterministic, fully observable games, to military simulations, which are typically stochastic and partially observable in nature. We develop, test, and comment on the results of two prototype implementations: one in a simple simulation environment with the objective of conserving friendly strength while depleting opposing forces, and the other focused on producing an optimal or near optimal assignment and schedule of aerial platforms against a set of missions with known values. Finally, we conclude by making recommendations for future implementations and applications in the COMBATXXI and JDAFS simulation environments, and suggest ways of addressing some of the computation challenges associated with Monte Carlo tree search and recursive simulation.

We present a scalable tree search planning algorithm for large multi-agent sequential decision problems that require dynamic collaboration. Teams of agents need to coordinate decisions in many domains, but naive approaches fail due to the exponential growth of the joint action space with the number of agents. We circumvent this complexity through an anytime approach that allows us to trade computation for approximation quality and also dynamically coordinate actions. Our algorithm comprises three elements: online planning with Monte Carlo Tree Search (MCTS), factored representations of local agent interactions with coordination graphs, and the iterative Max-Plus method for joint action selection. We evaluate our approach on the benchmark SysAdmin domain with static coordination graphs and achieve comparable performance with much lower computation cost than our MCTS baselines. We also introduce a multi-drone delivery domain with dynamic, i.e., state-dependent coordination graphs, and demonstrate how our approach scales to large problems on this domain that are intractable for other MCTS methods. We provide an open-source implementation of our algorithm at https://github.com/JuliaPOMDP/FactoredValueMCTS.jl.

We present a scalable tree search planning algorithm for large multi-agent sequential decision problems that require dynamic collaboration. Teams of agents need to coordinate decisions in many domains, but naive approaches fail due to the exponential growth of the joint action space with the number of agents. We circumvent this complexity through an anytime approach that allows us to trade computation for approximation quality and also dynamically coordinate actions. Our algorithm comprises three elements: online planning with Monte Carlo Tree Search (MCTS), factored representations of local agent interactions with coordination graphs, and the iterative Max-Plus method for joint action selection. We evaluate our approach on the benchmark SysAdmin domain with static coordination graphs and achieve comparable performance with much lower computation cost than our MCTS baselines. We also introduce a multi-drone delivery domain with dynamic, i.e., state-dependent coordination graphs, and demonstrate how our approach scales to large problems on this domain that are intractable for other MCTS methods. We provide an open-source implementation of our algorithm at https://github.com/JuliaPOMDP/FactoredValueMCTS.jl.

We present a scalable tree search planning algorithm for large multi-agent sequential decision problems that require dynamic collaboration. Teams of agents need to coordinate decisions in many domains, but naive approaches fail due to the exponential growth of the joint action space with the number of agents. We circumvent this complexity through an approach that allows us to trade computation for approximation quality and dynamically coordinate actions. Our algorithm comprises three elements: online planning with Monte Carlo Tree Search (MCTS), factored representations of local agent interactions with coordination graphs, and the iterative Max-Plus method for joint action selection. We evaluate our approach on the benchmark SysAdmin domain with static coordination graphs and achieve comparable performance with much lower computation cost than our MCTS baselines. We also introduce a multi-drone delivery domain with dynamic coordination graphs, and demonstrate how our approach scales to large problems on this domain that are intractable for other MCTS methods. We provide an open-source implementation of our algorithm at https://github.com/JuliaPOMDP/FactoredValueMCTS.jl.

Inspired by the core idea of AlphaGo, we combine a neural network, which is trained by Adaptive Dynamic Programming (ADP), with Monte Carlo Tree Search (MCTS) algorithm for Gomoku. MCTS algorithm is based on Monte Carlo simulation method, which goes through lots of simulations and generates a game search tree. We rollout it and search the outcomes of the leaf nodes in the tree. As a result, we obtain the MCTS winning rate. The ADP and MCTS methods are used to estimate the winning rates respectively. We weight the two winning rates to select the action position with the maximum one. Experiment result shows that this method can effectively eliminate the neural network evaluation function's "short-sighted" defect. With our proposed method, the game's final prediction result is more accurate, and it outperforms the Gomoku with ADP algorithm.