Monte Carlo Tree Search Algorithm Research Articles

Too hot to print, too slow to handle; finding optimal path characteristics for WAAM

Wire + Arc Additive Manufacturing (WAAM) is recognized as a highly capable metal additive process. However, the quality of manufactured parts is often compromised by defects stemming from process-induced temperature fields and transient weld-bead shapes, largely due to suboptimal path planning. Recent efforts have predominantly focused on optimizing either the thermal aspects or the productivity of WAAM, but the interplay between these factors and the final geometric accuracy of the parts has not been thoroughly investigated. This study introduces an automated framework for path planning optimization in WAAM, effectively bridging the gap between temperature control, geometric accuracy, and productivity. The approach utilizes a reinforcement learning agent to generate paths that minimize deposition time. These paths are subsequently segmented and optimized for their temperature response using a Monte Carlo tree search algorithm, with the thermal effects efficiently approximated through a reduced-order model. The paper presents a comparative analysis of various deposition strategies, offering insights and recommendations to enhance both productivity and part quality in WAAM path planning.

Integrated Decision-Making Approach for Automated Diversion Runway Selection and Flight Path Generation

The automated in-flight airport diversion trajectory generation in case of emergency is, to this day, an open problem and a real need for general and civil aviation. This problem is often addressed by solving three subproblems independently in cascade: i) selection of diversion airport, runway, and approach mode; ii) obstacle-free path planning; and iii) flyable trajectory generation under given aircraft performance. This paper proposes an integrated decision-making approach for subproblems (i) and (ii) by performing multitarget minimum-risk path planning. The N-best runway candidates are first identified based on runway-risk criteria. Then the Monte Carlo tree search algorithm is applied to simultaneously search for obstacle-free paths to these N candidates while minimizing a global risk by aggregating the en route and runway risks. Unlike the cascaded approaches, the proposed integrated approach can evaluate the en route risk more precisely and efficiently and use it in deciding the most suitable diversion runway. Moreover, the planning difficulty is also reflected in the runway selection. The proposed approach is tested in simulation for 1000 randomly generated missions. The results are compared with those of cascaded approaches, demonstrating a higher success rate of feasible path generation in difficult mission configurations.

Data-driven dynamic police patrolling: An efficient Monte Carlo tree search

Crime is responsible for major financial losses and serious harm to the well-being of individuals, and, hence, a crucial task of police operations is effective patrolling. Yet, in existing decision models aimed at police operations, microscopic routing decisions from patrolling are not considered, and, furthermore, the objective is limited to surrogate metrics (e.g., response time) instead of crime prevention. In this paper, we thus formalize the decision problem of dynamic police patrolling as a Markov decision process that models microscopic routing decisions, so that the expected number of prevented crimes are maximized. We experimentally show that standard solution approaches for our decision problem are not scalable to real-world settings. As a remedy, we present a tailored and highly efficient Monte Carlo tree search algorithm. We then demonstrate our algorithm numerically using real-world crime data from Chicago and show that the decision-making by our algorithm offers significant improvements for crime prevention over patrolling tactics from current practice. Informed by our results, we finally discuss implications for improving the patrolling tactics in police operations.

Multi-task oriented team formation in online collaborative learning

Rational and personalized team formation can greatly enhance the efficiency of online collaborative learning. However, in the process of forming learning teams and completing learning tasks, online synchronous communication among team members is often overlooked, and the abilities of team members may change during task execution, which may have an effect on the completion of team tasks. To address these problems, this paper comprehensively considers the impact of learners’ online time and skill levels on team formation. We quantify the potential development space of learners’ skills and design optimization objectives accordingly based on the Zone of Proximal Development theory. Additionally, we propose a Near-Informer deep learning model to predict learners’ future online states, aiming to measure communication efficiency. Then, we propose a multi-task oriented multi-objective optimization team formation problem, and transform it into a single-objective one by using weighted aggregation of all objectives. After that, we present a Three-dimensional Monte Carlo Tree Search algorithm (TDMCTS). The experiments are conducted based on two real datasets. Compared with the baselines, our Near-informer model achieved the highest number of optimal predictions, with 22 out of 32. Besides, TDMCTS algorithm achieved more comprehensive team formation results, with average skill satisfaction degrees of 99.8% and 98.9% on two task sets, respectively, while satisfying the task requirements.

Computational design of residential units’ floor layout: A heuristic algorithm

Computer-aided intelligent design can improve the quality and efficiency of residential building design. However, the design conditions are diverse, and numerous combinations of spatial elements exist. We propose a heuristic algorithm—the combined Monte Carlo tree search (MCTS) and particle swarm optimization (PSO) algorithm (PSO-MCTS)—for the automated design of residential floor layouts. The MCTS algorithm deals with discrete variables (room locations). It considers architectural design experience and design standards during node pruning to compress the search space and simplify the computation. The PSO algorithm deals with continuous variables (room sizes) and accelerates the computing speed due to parallel processing. The PSO-MCTS algorithm achieves a good balance between computational efficiency and accuracy. It is suitable for various design conditions and requirements. The input parameters include the daylight façade, boundary scope, entrance location, room type and size, and relationships with adjacent rooms. The PSO-MCTS provides 100 %, 80 %, and 75 % better results than heuristic algorithms, designs by architects, and a state-of-the-art deep learning model, respectively.

Generative design for complex floorplans in high-rise residential buildings: A Monte Carlo tree search-based self-organizing multi-agent system (MCTS-MAS) solution

This research aims to develop an innovative generative design algorithm for complex floorplan layout designs in high-rise residential buildings. Unlike conventional methods, which treat such a task as an optimization problem, we approach it as an adaption problem to be solved by combining a self-organized multi-agent system (MAS) and Monte Carlo tree search (MCTS) algorithm. The approach involves two steps. The first is design constraints pre-processing, which comprises (a) initial user input, and (b) pre-processing for initial setup. The second is building plan generation, involving (a) perception, (b) planning, and (c) action modules. Multiple illustrative examples are explored to verify the proposed generative design algorithm in terms of robustness, efficiency, and scale. We discover that the algorithm can handle complex combinations of input constraints (e.g., number of rooms, room areas, adjacency connections, and building boundaries) and generate plausible floorplans in a robust, efficient way that can be scaled up to bigger, more complex floorplan generative design problems. With proper adaptation, the algorithm can potentially be applied to other building types, such as office or hospital buildings.

PPB-MCTS: A novel distributed-memory parallel partial-backpropagation Monte Carlo tree search algorithm

Monte-Carlo Tree Search (MCTS) is an adaptive and heuristic tree-search algorithm designed to uncover sub-optimal actions at each decision-making point. This method progressively constructs a search tree by gathering samples throughout its execution. Predominantly applied within the realm of gaming, MCTS has exhibited exceptional achievements. Additionally, it has displayed promising outcomes when employed to solve NP-hard combinatorial optimization problems. MCTS has been adapted for distributed-memory parallel platforms. The primary challenges associated with distributed-memory parallel MCTS are the substantial communication overhead and the necessity to balance the computational load among various processes. In this work, we introduce a novel distributed-memory parallel MCTS algorithm with partial backpropagations, referred to as Parallel Partial-Backpropagation MCTS (PPB-MCTS). Our design approach aims to significantly reduce the communication overhead while maintaining, or even slightly improving, the performance in the context of combinatorial optimization problems. To address the communication overhead challenge, we propose a strategy involving transmitting an additional backpropagation message. This strategy avoids attaching an information table to the communication messages exchanged by the processes, thus reducing the communication overhead. Furthermore, this approach contributes to enhancing the decision-making accuracy during the selection phase. The load balancing issue is also effectively addressed by implementing a shared transposition table among the parallel processes. Furthermore, we introduce two primary methods for managing duplicate states within distributed-memory parallel MCTS, drawing upon techniques utilized in addressing duplicate states within sequential MCTS. Duplicate states can transform the conventional search tree into a Directed Acyclic Graph (DAG). To evaluate the performance of our proposed parallel algorithm, we conduct an extensive series of experiments on solving instances of the Job-Shop Scheduling Problem (JSSP) and the Weighted Set-Cover Problem (WSCP). These problems are recognized for their complexity and classified as NP-hard combinatorial optimization problems with considerable relevance within industrial applications. The experiments are performed on a cluster of computers with many cores. The empirical results highlight the enhanced scalability of our algorithm compared to that of the existing distributed-memory parallel MCTS algorithms. As the number of processes increases, our algorithm demonstrates increased rollout efficiency while maintaining an improved load balance across processes.

Stateful black-box fuzzing for encryption protocols and its application in IPsec

Owing to the rapid development of information security technology, the security analysis of encryption protocols has received widespread attention. In this paper, we propose a stateful black-box encryption protocol fuzzing method to analyze the security of real-world black-box encryption protocol programs and devices. This method does not rely on the source code but uses captured packets as input, performs state selection based on a Monte Carlo tree search algorithm, and processes the encryption/decryption conversion of mutant test cases based on the intermediate mapper. It sends test cases and collects responses in interactive communication with the tested program and dynamically optimizes the corpus based on the collected state information. Based on this method, we develop SBEPFuzz and primarily analyze IPsec. We evaluate SBEPFuzz on six widely used IPsec implementations. The experimental results show that SBEPFuzz achieves higher code coverage, and can discover more protocol state sequences and vulnerabilities. Furthermore, we discover four anomalies, including malformed packets triggering service crashes and abnormal interactions leading to plaintext ID leakage, which also reflect the differences in details among different implementations.

MoLPC2: improved prediction of large protein complex structures and stoichiometry using Monte Carlo Tree Search and AlphaFold2.

Today, the prediction of structures of large protein complexes solely from their sequence information requires prior knowledge of the stoichiometry of the complex. To address this challenge, we have enhanced the Monte Carlo Tree Search algorithms in MoLPC to enable the assembly of protein complexes while simultaneously predicting their stoichiometry. In MoLPC2, we have improved the predictions by allowing sampling alternative AlphaFold predictions. Using MoLPC2, we accurately predicted the structures of 50 out of 175 nonredundant protein complexes (TM-score ≥ 0.8) without knowing the stoichiometry. MoLPC2 provides new opportunities for predicting protein complex structures without stoichiometry information. MoLPC2 is freely available at https://github.com/hychim/molpc2. A notebook is also available from the repository for easy use.

The application of chessboard game based on integrated learning and UCT algorithm in mental health and emotional regulation

Go plays a very prominent role in quality education and mental health education. Go can cultivate character, make friends, improve interpersonal relationships, and promote mental health. To apply it to mental health and emotional regulation, the AlphaGo series algorithms are analyzed. To address the unstable deep reinforcement learning training, low training data utilization, model sensitivity to hyper-parameters, and heavy parameter tuning burden, the algorithm is optimized by combining ensemble learning and advanced deep reinforcement learning. Then, the weighted voting method is applied to the selection step of sub nodes in the Monte Carlo tree search process. At the same time, asynchronous parallelization improvements are implemented to improve the efficiency of the Monte Carlo tree search algorithm. Finally, a board game based on ensemble learning and improved Monte Carlo tree search algorithm is obtained. Through experimental analysis, the average winning rate of the system was 93.36 %. The number of users who were very satisfied with the game was 83.96 %. After intervention, the depression and anxiety scores of the experimental group students were significantly lower than the control group (P < 0.05). The designed game can effectively alleviate user anxiety and depression, providing new ideas for the psychotherapy.

Monte Carlo tree search control scheme for multibody dynamics applications

There is considerable interest in applying reinforcement learning (RL) to improve machine control across multiple industries, and the automotive industry is one of the prime examples. Monte Carlo Tree Search (MCTS) has emerged and proven powerful in decision-making games, even without understanding the rules. In this study, multibody system dynamics (MSD) control is first modeled as a Markov Decision Process and solved with Monte Carlo Tree Search. Based on randomized search space exploration, the MCTS framework builds a selective search tree by repeatedly applying a Monte Carlo rollout at each child node. However, without a library of available choices, deciding among the many possibilities for agent parameters can be intimidating. In addition, the MCTS poses a significant challenge for searching due to the large branching factor. This challenge is typically overcome by appropriate parameter design, search guiding, action reduction, parallelization, and early termination. To address these shortcomings, the overarching goal of this study is to provide needed insight into inverted pendulum controls via vanilla and modified MCTS agents, respectively. A series of reward functions are well-designed according to the control goal, which maps a specific distribution shape of reward bonus and guides the MCTS-based control to maintain the upright position. Numerical examples show that the reward-modified MCTS algorithms significantly improve the control performance and robustness of the default choice of a constant reward that constitutes the vanilla MCTS. The exponentially decaying reward functions perform better than the constant value or polynomial reward functions. Moreover, the exploitation vs. exploration trade-off and discount parameters are carefully tested. The study’s results can guide the research of RL-based MSD users.

Application of Monte Carlo Tree Search algorithm in Go playing

The reputation of Monte Carlo Tree Search (MCTS) algorithm is built upon its ability to handle a large number of options and make optimal decisions and detailed plans in situations with limited information. It was initially applied in the game of Go, a traditional and complex combinatorial strategy game, which severely challenged the efficiency of traditional brute-force algorithms that relied on simulating every node and branch in the search tree. The success of AlphaGo, developed by Google, represents a significant advancement in artificial intelligence technology, utilizing deep learning networks and MCTS techniques. In this article, we analyze the limitations of traditional brute-force approaches in Go playing and compare them with the application of MCTS, which overcomes these limitations through the iterative process of four steps: selection, expansion, simulation, and back-propagation. The discussion covers the application of MCTS in the game of Go. Besides, we evaluate the advantages and shortcomings of the MCTS algorithm. The article concludes with a summary and prospects for future research.

Energy efficient deployment of aerial base stations for mobile users in multi-hop UAV networks

Unmanned aerial vehicles (UAVs) are popularly considered as aerial base stations in a Low-Altitude Platform (LAP) to provide wireless connections to ground users in disaster and digital divide situation. Most of previous studies have investigated energy efficient UAV deployment only for the stationary ground users. Contrarily, we propose an on-line deployment algorithm for ground mobile users (GMUs) who are partially observable in the multi-UAV system. In the framework of Partially Observable Markov Decision Process (POMDP) with large state and action space, we use the Monte Carlo Tree Search (MCTS) algorithm enhanced by the Double Progressive Widening (DPW) and Deep Neural Network (DNN)-based guidance. For online learning of the DNN, simulated samples are used with Proximal Policy Optimization (PPO) that prevents excessive training for a particular belief. From experiment in urban environment, the proposed scheme shows better learning performance than the MCTS with rollout policy and faster convergence than training with environment samples. In addition, it optimizes dual objectives proportionally in a trade-off between energy saving and throughput including prediction error on GMU location.

MCTS with Refinement for Proposals Selection Games in Scene Understanding.

We propose a novel method applicable in many scene understanding problems that adapts the Monte Carlo Tree Search (MCTS) algorithm, originally designed to learn to play games of high-state complexity. From a generated pool of proposals, our method jointly selects and optimizes proposals that minimize the objective term. In our first application for floor plan reconstruction from point clouds, our method selects and refines the room proposals, modelled as 2D polygons, by optimizing on an objective function combining the fitness as predicted by a deep network and regularizing terms on the room shapes. We also introduce a novel differentiable method for rendering the polygonal shapes of these proposals. Our evaluations on the recent and challenging Structured3D and Floor-SP datasets show significant improvements over the state-of-the-art both in speed and quality of reconstructions, without imposing hard constraints nor assumptions on the floor plan configurations. In our second application, we extend our approach to reconstruct general 3D room layouts from a color image and obtain accurate room layouts. We also show that our differentiable renderer can easily be extended for rendering 3D planar polygons and polygon embeddings. Our method shows high performance on the Matterport3D-Layout dataset, without introducing hard constraints on room layout configurations.

Collaborative Cost Multi-Agent Decision-Making Algorithm with Factored-Value Monte Carlo Tree Search and Max-Plus

In this paper, we describe the Factored Value MCTS Hybrid Cost-Max-Plus algorithm, a collection of decision-making algorithms (centralized, decentralized, and hybrid) for a multi-agent system in a collaborative setting that considers action costs. Our proposed algorithm is made up of two steps. In the first step, each agent searches for the best individual actions with the lowest cost using the Monte Carlo Tree Search (MCTS) algorithm. Each agent’s most promising activities are chosen and presented to the team. The Hybrid Cost Max-Plus method is utilized for joint action selection in the second step. The Hybrid Cost Max-Plus algorithm improves the well-known centralized and distributed Max-Plus algorithm by incorporating the cost of actions in agent interactions. The Max-Plus algorithm employed the Coordination Graph framework, which exploits agent dependencies to decompose the global payoff function as the sum of local terms. In terms of the number of agents and their interactions, the suggested Factored Value MCTS-Hybrid Cost-Max-Plus method is online, anytime, distributed, and scalable. Our contribution competes with state-of-the-art methodologies and algorithms by leveraging the locality of agent interactions for planning and acting utilizing MCTS and Max-Plus algorithms.

Hypothesis selection with Monte Carlo tree search for feature-based simultaneous localization and mapping in non-static environments

A static world assumption is often used when considering the simultaneous localization and mapping (SLAM) problem. In reality, especially when long-term autonomy is the objective, this is not a valid assumption. This paper studies a scenario where landmarks can occupy multiple discrete positions at different points in time, where each possible position is added to a multi-hypothesis map representation. A selector-mixture distribution is introduced and used in the observation model. Each landmark position hypothesis is associated with one component in the mixture. The landmark movements are modeled by a discrete Markov chain and the Monte Carlo tree search algorithm is suggested to be used as component selector. The non-static environment model is further incorporated into the factor graph formulation of the SLAM problem and is solved by iterating between estimating discrete variables with a component selector and optimizing continuous variables with an efficient state-of-the-art nonlinear least squares SLAM solver. The proposed non-static SLAM system is validated in numerical simulation and with a publicly available dataset by showing that a non-static environment can successfully be navigated.

Timing-Aware Qubit Mapping and Gate Scheduling Adapted to Neutral Atom Quantum Computing

As a less developed but potential quantum technology, neutral atoms (NA) can provide advantages including higher qubit connectivity, longer-range interactions, and much more native multi-control gates than superconductivity. Long-range interactions, however, prevent parallelism of interacting qubit pairs with their surrounding restriction zones. Therefore, the quantum program cannot be run directly on a neutral atom quantum computer (NAQC) unless compiled. The recent compiling study of NAQC applies simple layer-by-layer scheduling and does not consider the difference in gate duration. To address the above issues, we focus on the qubit mapping and quantum gate scheduling () problem of quantum circuits to meet hardware constraints of superconductivity and even neutral atoms. Our goal is to shorten the execution time to mitigate decoherence noise. We propose a block-game-like abstraction mechanism which is an abstract model of the problem and aware of the duration difference of gates and several other NA characteristics. Based on the abstraction, we propose a heuristic greedy algorithm (HGA) to solve the problem efficiently. To further speed up the execution of the circuit, we embed HGA into a Monte Carlo tree search (MCTS) framework to solve the problem, which consumes more compiling time but achieves a better result. Comparing our MCTS algorithm with the only recent algorithm for NAQC, the average speedup ratio is 1.75× for several quantum circuits collected from RevLib, and 1.18× for circuits from Qiskit Lib.

Overview of selected reinforcement learning solutions to several game theory problems

This paper collects several applications of reinforcement learning in solving some problems related to game theory. The methods were selected to possibly show variety of problems and approaches. Selections includes Thompson Sampling, Q-learning, DQN and AlphaGo Zero using Monte Carlo Tree Search algorithm. Paper attempts to show intuition behind proposed algorithms with shallow explaining of technical details. This approach aims at presenting overview of the topic without assuming deep knowledge about statistics and artificial intelligence.

A Conflict Resolution Strategy at a Taxiway Intersection by Combining a Monte Carlo Tree Search with Prior Knowledge

With the escalating complexity of surface operations at large airports, the conflict risk for aircraft taxiing has correspondingly increased. Usually, the Air Traffic Controllers (ATCOs) generate route, speed and holding instructions to resolve conflicts. In this paper, we introduce a conflict resolution framework that incorporates prior knowledge by integrating a Multi-Layer Perceptron (MLP) neural network into the Monte Carlo Tree Search (MCTS) approach. The neural network is trained to learn the allocation strategy for waiting time extracted from actual aircraft taxiing trajectory data. Subsequently, the action probability distribution generated with the neural network is embedded into the MCTS algorithm as a heuristic evaluation function to guide the search process in finding the optimal conflict resolution strategy. Experimental results show that the average conflict resolution rate is 96.8% in different conflict scenarios, and the taxiing time required to resolve conflicts is reduced by an average of 42.77% compared to the taxiing time in actual airport surface operations.

A review on the application of reinforcement learning to the 2048 game

In light of the rapid development of Artificial Intelligence (AI), this paper pries into the application of reinforcement learning on the 2048 game which has important implications on the algorithms ability to solve various real-life problems including but not limited to robotics, auto-driving, and financial trading. The paper summarizes the possible usage of the 2048 model in Q-learning, convolutional neural networks, genetic, and Monte Carlo tree search algorithms. Then the paper scopes into the current progress and methods used in solving the 2048 game with reinforcement learning. Starting off with Q-learning, its implementation and limitations, such as large state space, sparse rewards, limited exploration, computational complexity, and a lack of human-like intuition, are illustrated. Deep Q-learning handles the problem of large state space from Q-learning. However, most of the disadvantages that are found in Q-learning are shared by Deep Q-learning and hence should become the focus of future research on using reinforcement learning to solve 2048.