Monte Carlo Tree Search Algorithm Research Articles

MoLPC2: Improved prediction of large protein complex structures and stoichiometry using monte carlo tree search and AlphaFold2.

Today, the prediction of structures of large protein complexes solely from their sequence information requires prior knowledge of the stoichiometry of the complex. To address this challenge, we have enhanced the Monte Carlo Tree Search algorithms in MoLPC to enable the assembly of protein complexes while simultaneously predicting their stoichiometry. In MoLPC2, we have improved the predictions by allowing sampling alternative AlphaFold predictions. Using MoLPC2, we accurately predicted the structures of 50 out of 175 non-redundant protein complexes (TM-score > = 0.8) without knowing the stoichiometry. MoLPC2 provides new opportunities for predicting protein complex structures without stoichiometry information. MoLPC2 is freely available at https://github.com/hychim/molpc2. A notebook is also available from the repository for easy use. Supplementary data are available at Bioinformatics online.

The application of chessboard game based on integrated learning and UCT algorithm in mental health and emotional regulation

Go plays a very prominent role in quality education and mental health education. Go can cultivate character, make friends, improve interpersonal relationships, and promote mental health. To apply it to mental health and emotional regulation, the AlphaGo series algorithms are analyzed. To address the unstable deep reinforcement learning training, low training data utilization, model sensitivity to hyper-parameters, and heavy parameter tuning burden, the algorithm is optimized by combining ensemble learning and advanced deep reinforcement learning. Then, the weighted voting method is applied to the selection step of sub nodes in the Monte Carlo tree search process. At the same time, asynchronous parallelization improvements are implemented to improve the efficiency of the Monte Carlo tree search algorithm. Finally, a board game based on ensemble learning and improved Monte Carlo tree search algorithm is obtained. Through experimental analysis, the average winning rate of the system was 93.36 %. The number of users who were very satisfied with the game was 83.96 %. After intervention, the depression and anxiety scores of the experimental group students were significantly lower than the control group (P < 0.05). The designed game can effectively alleviate user anxiety and depression, providing new ideas for the psychotherapy.

Monte Carlo tree search control scheme for multibody dynamics applications

There is considerable interest in applying reinforcement learning (RL) to improve machine control across multiple industries, and the automotive industry is one of the prime examples. Monte Carlo Tree Search (MCTS) has emerged and proven powerful in decision-making games, even without understanding the rules. In this study, multibody system dynamics (MSD) control is first modeled as a Markov Decision Process and solved with Monte Carlo Tree Search. Based on randomized search space exploration, the MCTS framework builds a selective search tree by repeatedly applying a Monte Carlo rollout at each child node. However, without a library of available choices, deciding among the many possibilities for agent parameters can be intimidating. In addition, the MCTS poses a significant challenge for searching due to the large branching factor. This challenge is typically overcome by appropriate parameter design, search guiding, action reduction, parallelization, and early termination. To address these shortcomings, the overarching goal of this study is to provide needed insight into inverted pendulum controls via vanilla and modified MCTS agents, respectively. A series of reward functions are well-designed according to the control goal, which maps a specific distribution shape of reward bonus and guides the MCTS-based control to maintain the upright position. Numerical examples show that the reward-modified MCTS algorithms significantly improve the control performance and robustness of the default choice of a constant reward that constitutes the vanilla MCTS. The exponentially decaying reward functions perform better than the constant value or polynomial reward functions. Moreover, the exploitation vs. exploration trade-off and discount parameters are carefully tested. The study’s results can guide the research of RL-based MSD users.

Application of Monte Carlo Tree Search algorithm in Go playing

The reputation of Monte Carlo Tree Search (MCTS) algorithm is built upon its ability to handle a large number of options and make optimal decisions and detailed plans in situations with limited information. It was initially applied in the game of Go, a traditional and complex combinatorial strategy game, which severely challenged the efficiency of traditional brute-force algorithms that relied on simulating every node and branch in the search tree. The success of AlphaGo, developed by Google, represents a significant advancement in artificial intelligence technology, utilizing deep learning networks and MCTS techniques. In this article, we analyze the limitations of traditional brute-force approaches in Go playing and compare them with the application of MCTS, which overcomes these limitations through the iterative process of four steps: selection, expansion, simulation, and back-propagation. The discussion covers the application of MCTS in the game of Go. Besides, we evaluate the advantages and shortcomings of the MCTS algorithm. The article concludes with a summary and prospects for future research.

Energy efficient deployment of aerial base stations for mobile users in multi-hop UAV networks

Unmanned aerial vehicles (UAVs) are popularly considered as aerial base stations in a Low-Altitude Platform (LAP) to provide wireless connections to ground users in disaster and digital divide situation. Most of previous studies have investigated energy efficient UAV deployment only for the stationary ground users. Contrarily, we propose an on-line deployment algorithm for ground mobile users (GMUs) who are partially observable in the multi-UAV system. In the framework of Partially Observable Markov Decision Process (POMDP) with large state and action space, we use the Monte Carlo Tree Search (MCTS) algorithm enhanced by the Double Progressive Widening (DPW) and Deep Neural Network (DNN)-based guidance. For online learning of the DNN, simulated samples are used with Proximal Policy Optimization (PPO) that prevents excessive training for a particular belief. From experiment in urban environment, the proposed scheme shows better learning performance than the MCTS with rollout policy and faster convergence than training with environment samples. In addition, it optimizes dual objectives proportionally in a trade-off between energy saving and throughput including prediction error on GMU location.

MCTS with Refinement for Proposals Selection Games in Scene Understanding.

We propose a novel method applicable in many scene understanding problems that adapts the Monte Carlo Tree Search (MCTS) algorithm, originally designed to learn to play games of high-state complexity. From a generated pool of proposals, our method jointly selects and optimizes proposals that minimize the objective term. In our first application for floor plan reconstruction from point clouds, our method selects and refines the room proposals, modelled as 2D polygons, by optimizing on an objective function combining the fitness as predicted by a deep network and regularizing terms on the room shapes. We also introduce a novel differentiable method for rendering the polygonal shapes of these proposals. Our evaluations on the recent and challenging Structured3D and Floor-SP datasets show significant improvements over the state-of-the-art both in speed and quality of reconstructions, without imposing hard constraints nor assumptions on the floor plan configurations. In our second application, we extend our approach to reconstruct general 3D room layouts from a color image and obtain accurate room layouts. We also show that our differentiable renderer can easily be extended for rendering 3D planar polygons and polygon embeddings. Our method shows high performance on the Matterport3D-Layout dataset, without introducing hard constraints on room layout configurations.

Collaborative Cost Multi-Agent Decision-Making Algorithm with Factored-Value Monte Carlo Tree Search and Max-Plus

In this paper, we describe the Factored Value MCTS Hybrid Cost-Max-Plus algorithm, a collection of decision-making algorithms (centralized, decentralized, and hybrid) for a multi-agent system in a collaborative setting that considers action costs. Our proposed algorithm is made up of two steps. In the first step, each agent searches for the best individual actions with the lowest cost using the Monte Carlo Tree Search (MCTS) algorithm. Each agent’s most promising activities are chosen and presented to the team. The Hybrid Cost Max-Plus method is utilized for joint action selection in the second step. The Hybrid Cost Max-Plus algorithm improves the well-known centralized and distributed Max-Plus algorithm by incorporating the cost of actions in agent interactions. The Max-Plus algorithm employed the Coordination Graph framework, which exploits agent dependencies to decompose the global payoff function as the sum of local terms. In terms of the number of agents and their interactions, the suggested Factored Value MCTS-Hybrid Cost-Max-Plus method is online, anytime, distributed, and scalable. Our contribution competes with state-of-the-art methodologies and algorithms by leveraging the locality of agent interactions for planning and acting utilizing MCTS and Max-Plus algorithms.

Hypothesis selection with Monte Carlo tree search for feature-based simultaneous localization and mapping in non-static environments

A static world assumption is often used when considering the simultaneous localization and mapping (SLAM) problem. In reality, especially when long-term autonomy is the objective, this is not a valid assumption. This paper studies a scenario where landmarks can occupy multiple discrete positions at different points in time, where each possible position is added to a multi-hypothesis map representation. A selector-mixture distribution is introduced and used in the observation model. Each landmark position hypothesis is associated with one component in the mixture. The landmark movements are modeled by a discrete Markov chain and the Monte Carlo tree search algorithm is suggested to be used as component selector. The non-static environment model is further incorporated into the factor graph formulation of the SLAM problem and is solved by iterating between estimating discrete variables with a component selector and optimizing continuous variables with an efficient state-of-the-art nonlinear least squares SLAM solver. The proposed non-static SLAM system is validated in numerical simulation and with a publicly available dataset by showing that a non-static environment can successfully be navigated.

Timing-Aware Qubit Mapping and Gate Scheduling Adapted to Neutral Atom Quantum Computing

As a less developed but potential quantum technology, neutral atoms (NA) can provide advantages including higher qubit connectivity, longer-range interactions, and much more native multi-control gates than superconductivity. Long-range interactions, however, prevent parallelism of interacting qubit pairs with their surrounding restriction zones. Therefore, the quantum program cannot be run directly on a neutral atom quantum computer (NAQC) unless compiled. The recent compiling study of NAQC applies simple layer-by-layer scheduling and does not consider the difference in gate duration. To address the above issues, we focus on the qubit mapping and quantum gate scheduling () problem of quantum circuits to meet hardware constraints of superconductivity and even neutral atoms. Our goal is to shorten the execution time to mitigate decoherence noise. We propose a block-game-like abstraction mechanism which is an abstract model of the problem and aware of the duration difference of gates and several other NA characteristics. Based on the abstraction, we propose a heuristic greedy algorithm (HGA) to solve the problem efficiently. To further speed up the execution of the circuit, we embed HGA into a Monte Carlo tree search (MCTS) framework to solve the problem, which consumes more compiling time but achieves a better result. Comparing our MCTS algorithm with the only recent algorithm for NAQC, the average speedup ratio is 1.75× for several quantum circuits collected from RevLib, and 1.18× for circuits from Qiskit Lib.

Overview of selected reinforcement learning solutions to several game theory problems

This paper collects several applications of reinforcement learning in solving some problems related to game theory. The methods were selected to possibly show variety of problems and approaches. Selections includes Thompson Sampling, Q-learning, DQN and AlphaGo Zero using Monte Carlo Tree Search algorithm. Paper attempts to show intuition behind proposed algorithms with shallow explaining of technical details. This approach aims at presenting overview of the topic without assuming deep knowledge about statistics and artificial intelligence.

A Conflict Resolution Strategy at a Taxiway Intersection by Combining a Monte Carlo Tree Search with Prior Knowledge

With the escalating complexity of surface operations at large airports, the conflict risk for aircraft taxiing has correspondingly increased. Usually, the Air Traffic Controllers (ATCOs) generate route, speed and holding instructions to resolve conflicts. In this paper, we introduce a conflict resolution framework that incorporates prior knowledge by integrating a Multi-Layer Perceptron (MLP) neural network into the Monte Carlo Tree Search (MCTS) approach. The neural network is trained to learn the allocation strategy for waiting time extracted from actual aircraft taxiing trajectory data. Subsequently, the action probability distribution generated with the neural network is embedded into the MCTS algorithm as a heuristic evaluation function to guide the search process in finding the optimal conflict resolution strategy. Experimental results show that the average conflict resolution rate is 96.8% in different conflict scenarios, and the taxiing time required to resolve conflicts is reduced by an average of 42.77% compared to the taxiing time in actual airport surface operations.

A review on the application of reinforcement learning to the 2048 game

In light of the rapid development of Artificial Intelligence (AI), this paper pries into the application of reinforcement learning on the 2048 game which has important implications on the algorithms ability to solve various real-life problems including but not limited to robotics, auto-driving, and financial trading. The paper summarizes the possible usage of the 2048 model in Q-learning, convolutional neural networks, genetic, and Monte Carlo tree search algorithms. Then the paper scopes into the current progress and methods used in solving the 2048 game with reinforcement learning. Starting off with Q-learning, its implementation and limitations, such as large state space, sparse rewards, limited exploration, computational complexity, and a lack of human-like intuition, are illustrated. Deep Q-learning handles the problem of large state space from Q-learning. However, most of the disadvantages that are found in Q-learning are shared by Deep Q-learning and hence should become the focus of future research on using reinforcement learning to solve 2048.

A Continuous Action Space Tree search for INverse desiGn (CASTING) framework for materials discovery

Material properties share an intrinsic relationship with their structural attributes, making inverse design approaches crucial for discovering new materials with desired functionalities. Reinforcement Learning (RL) approaches are emerging as powerful inverse design tools, often functioning in discrete action spaces. This constrains their application in materials design problems, which involve continuous search spaces. Here, we introduce an RL-based framework CASTING (Continuous Action Space Tree Search for inverse design), that employs a decision tree-based Monte Carlo Tree Search (MCTS) algorithm with continuous space adaptation through modified policies and sampling. Using representative examples like Silver (Ag) for metals, Carbon (C) for covalent systems, and multicomponent systems such as graphane, boron nitride, and complex correlated oxides, we showcase its accuracy, convergence speed, and scalability in materials discovery and design. Furthermore, with the inverse design of super-hard Carbon phases, we demonstrate CASTING’s utility in discovering metastable phases tailored to user-defined target properties and preferences.

The importance of V2X simulation: An in-depth comparison of intersection control algorithms using a high-fidelity communication simulation

The rapid evolution of vehicle-to-vehicle and vehicle-to-infrastructure communication opens doors to various control algorithms, one particular domain being intersection control. Several researchers have proposed communication-based intersection control algorithms omitting traditional traffic lights with the promise of enhanced throughput and improved traffic safety. On the other hand, the majority of these algorithms ignore the uncertainties and delays of communication and overlook new cyberattack vectors that are opened by connected traffic. The present study employs a highly detailed simulation of vehicular communication to highlight the sensitivity of different autonomous intersection control algorithms (both centralized and decentralized) to communication-related imperfections. The paper investigates five different algorithms borrowed from the literature in a comparative way. This research focuses on traffic-related parameters such as average speed, occupancy, and network throughput while also analyzing communication-related parameters (e.g., packet loss, computational demand) and considering possible attack vectors. Simulation results suggest that even the simplest control logic is sensitive to communication failures, degrading intersection throughput below traditional intersection control or even compromising traffic safety. For centralized algorithms, in the presence of noise, average speeds drop significantly, suggesting reduced intersection throughput and even gridlock for the First Come First Serve algorithm. Decentralized algorithms can be heavily affected by incoming message orders, which can lead to dangerous situations. During the simulations, multiple collisions were registered for the Monte-Carlo Tree Search algorithm. More complex algorithms that rely on accurate prediction on vehicle trajectories are more sensitive to noise and even produce accidents in the simulation. In conclusion, regardless of control architecture, the evaluated algorithms require additional fallback solutions and redundancies to retain traffic safety.

Evolving the MCTS Upper Confidence Bounds for Trees Using a Semantic-Inspired Evolutionary Algorithm in the Game of Carcassonne

Monte Carlo Tree Search (MCTS) is a sampling best-first method to search for optimal decisions. The success of MCTS depends heavily on how the tree is built and the selection process plays a fundamental role in this. One particular selection mechanism that has proved to be reliable is based on the Upper Confidence Bounds for Trees (UCT). The UCT attempts to balance exploration and exploitation by considering the values stored in the statistical tree of the MCTS. However, some tuning of the MCTS UCT is necessary for this to work well. In this work, we use Evolutionary Algorithms (EAs) to evolve mathematical expressions with the goal to substitute the UCT formula and use the evolved expressions in MCTS. More specifically, we evolve expressions by means of our proposed Semantic-inspired Evolutionary Algorithm in MCTS approach (SIEA-MCTS). This is inspired by semantics in Genetic Programming (GP), where the use of fitness cases is seen as a requirement to be adopted in GP. Fitness cases are normally used to determine the fitness of individuals and can be used to compute the semantic similarity (or dissimilarity) of individuals. However, fitness cases are not available in MCTS. We extend this notion by using multiple reward values from MCTS that allow us to determine both the fitness of an individual and its semantics. By doing so, we show how SIEA-MCTS is able to successfully evolve mathematical expressions that yield better or competitive results compared to UCT without the need of tuning these evolved expressions. We compare the performance of the proposed SIEA-MCTS against MCTS algorithms, MCTS Rapid Action Value Estimation algorithms, three variants of the *-minimax family of algorithms, a random controller and two more EA approaches. We consistently show how SIEA-MCTS outperforms most of these intelligent controllers in the challenging game of Carcassonne, whose state-space complexity is, approx., 10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{40}$</tex-math></inline-formula> .

Monte Carlo Planning in Hybrid Belief POMDPs

Real-world problems often require reasoning about hybrid beliefs, over both discrete and continuous random vari- ables. Yet, such a setting has hardly been investigated in the context of planning. Moreover, existing online Partially Ob- servable Markov Decision Processes (POMDPs) solvers do not support hybrid beliefs directly. In particular, these solvers do not address the added computational burden due to an increasing number of hypotheses with the planning horizon, which can grow exponentially. As part of this work, we present a novel algorithm, Hybrid Belief Monte Carlo Planning (HB-MCP) that utilizes the Monte Carlo Tree Search (MCTS) algorithm to solve a POMDP while maintaining a hybrid belief. We illustrate how the upper confidence bound (UCB) exploration bonus can be leveraged to guide the growth of hypotheses trees alongside the belief trees. We then evaluate our approach in highly aliased simulated environments where unresolved data association leads to multi-modal belief hypotheses.

Reinforcement learning based hybrid bond-order coarse-grained interatomic potentials for exploring mesoscale aggregation in liquid-liquid mixtures.

Exploring mesoscopic physical phenomena has always been a challenge for brute-force all-atom molecular dynamics simulations. Although recent advances in computing hardware have improved the accessible length scales, reaching mesoscopic timescales is still a significant bottleneck. Coarse-graining of all-atom models allows robust investigation of mesoscale physics with a reduced spatial and temporal resolution but preserves desired structural features of molecules, unlike continuum-based methods. Here, we present a hybrid bond-order coarse-grained forcefield (HyCG) for modeling mesoscale aggregation phenomena in liquid-liquid mixtures. The intuitive hybrid functional form of the potential offers interpretability to our model, unlike many machine learning based interatomic potentials. We parameterize the potential with the continuous action Monte Carlo Tree Search (cMCTS) algorithm, a reinforcement learning (RL) based global optimizing scheme, using training data from all-atom simulations. The resulting RL-HyCG correctly describes mesoscale critical fluctuations in binary liquid-liquid extraction systems. cMCTS, the RL algorithm, accurately captures the mean behavior of various geometrical properties of the molecule of interest, which were excluded from the training set. The developed potential model along with the RL-based training workflow could be applied to explore a variety of other mesoscale physical phenomena that are typically inaccessible to all-atom molecular dynamics simulations.

A self-learning Monte Carlo tree search algorithm for robot path planning.

This paper proposes a self-learning Monte Carlo tree search algorithm (SL-MCTS), which has the ability to continuously improve its problem-solving ability in single-player scenarios. SL-MCTS combines the MCTS algorithm with a two-branch neural network (PV-Network). The MCTS architecture can balance the search for exploration and exploitation. PV-Network replaces the rollout process of MCTS and predicts the promising search direction and the value of nodes, which increases the MCTS convergence speed and search efficiency. The paper proposes an effective method to assess the trajectory of the current model during the self-learning process by comparing the performance of the current model with that of its best-performing historical model. Additionally, this method can encourage SL-MCTS to generate optimal solutions during the self-learning process. We evaluate the performance of SL-MCTS on the robot path planning scenario. The experimental results show that the performance of SL-MCTS is far superior to the traditional MCTS and single-player MCTS algorithms in terms of path quality and time consumption, especially its time consumption is half less than that of the traditional MCTS algorithms. SL-MCTS also performs comparably to other iterative-based search algorithms designed specifically for path planning tasks.

Development of an MCTS Model for Hydrogen Production Optimisation

Hydrogen has the potential to revolutionize the energy industry due to its clean-burning and versatile properties. It is the most abundant element in the universe and can be produced through a variety of methods, including electrolysis. The widespread adoption of hydrogen faces various challenges, including the high cost of production; thus, it is important to optimise the production processes. This research focuses on development of models for hydrogen production optimisation based on various external factors and parameters. Models based on electricity prices are developed and compared between different market situations. To run hydrogen production more effectively, it is required to use renewable energy sources for the production process. Adding the solar power component to the economic evaluation model outcome is more positive. The Monte Carlo tree search (MCTS) algorithm is adapted to effectively control the electrolysis process. MCTS schedule optimization was performed for a 24 h time horizon applying two time-resolution settings—1 h and 15 min. The results demonstrate the potential of the MCTS algorithm for finding good schedules for water electrolyser devices by taking into account variable environmental factors. Whereas the MCTS with a 15 min resolution ensures mathematically better results, it requires more computational power to solve the decision tree.

AlphaRoute: Large-Scale Coordinated Route Planning via Monte Carlo Tree Search

This paper proposes AlphaRoute, an AlphaGo inspired algorithm for coordinating large-scale routes, built upon graph attention reinforcement learning and Monte Carlo Tree Search (MCTS). We first partition the road network into regions and model large-scale coordinated route planning as a Markov game, where each partitioned region is treated as a player instead of each driver. Then, AlphaRoute applies a bilevel optimization framework, consisting of several region planners and a global planner, where the region planner coordinates the route choices for vehicles located in the region and generates several strategies, and the global planner evaluates the combination of strategies. AlphaRoute is built on graph attention network for evaluating each state and MCTS algorithm for dynamically visiting and simulating the future state for narrowing down the search space. AlphaRoute is capable of 1) bridging user fairness and system efficiency, 2) achieving higher search efficiency by alleviating the curse of dimensionality problems, and 3) making an effective and informed route planning by simulating over the future to capture traffic dynamics. Comprehensive experiments are conducted on two real-world road networks as compared with several baselines to evaluate the performance, and results show that AlphaRoute achieves the lowest travel time, and is efficient and effective for coordinating large-scale routes and alleviating the traffic congestion problem. The code will be publicly available.