Monte Carlo Tree Research Articles

Overview of selected reinforcement learning solutions to several game theory problems

This paper collects several applications of reinforcement learning in solving some problems related to game theory. The methods were selected to possibly show variety of problems and approaches. Selections includes Thompson Sampling, Q-learning, DQN and AlphaGo Zero using Monte Carlo Tree Search algorithm. Paper attempts to show intuition behind proposed algorithms with shallow explaining of technical details. This approach aims at presenting overview of the topic without assuming deep knowledge about statistics and artificial intelligence.

A suggestion-based hierarchical energy efficient control framework for connected and automated vehicles in mixed stream urban road networks

Connected and automated vehicles (CAVs) and their advanced control systems have the ability to improve energy efficiency and mobility of the transportation system, but hundred percent market penetration of CAVs will not happen in the near future. CAVs are thus expected to share the road with human-driven vehicles (HDVs), which can act in suboptimal manners. Since vehicles on road constitute an interactive system, where actions of one vehicle influence the actions of its surrounding vehicles, suboptimal behavior of HDVs affect all their surrounding vehicles including the CAVs. This paper thus focuses on the development of a suggestion-based control framework for CAVs in interactive mixed traffic environments. Our mixed traffic environment includes both CAVs and connected HDVs that have vehicle-to-vehicle (V2V) communication capabilities. In this suggestion-based control framework, a CAV is considered to provide velocity and lane-change suggestions to multiple surrounding HDVs to follow. These suggestions are the velocities and lane-change actions the CAV wants the HDVs to follow to improve its own and the group’s energy efficiency and mobility. To take into account the interactive nature of the environment, the CAV is considered to jointly plan its own trajectory, evaluate the suggestions to the surrounding HDVs, and predict the actions of its surrounding vehicles. The joint planning requires solving the problem in joint state- and action-space, and in this paper, we develop a Monte Carlo Tree Search (MCTS)-based trajectory planning approach for the CAV. Since the joint action- and state-space grows exponentially with the number of agents and can be computationally expensive, we propose an adaptive action-space by pruning the actions resulting in unsafe trajectories for each agent. The trajectory planning approach is followed by a low-level model predictive control (MPC)-based motion controller, which aims at tracking the reference trajectory in an optimal fashion. Simulation studies show the proposed control strategy’s efficacy compared with existing baseline methods. Simulation studies show improvements in fuel efficiency ranging from 4% to 50% for HDVs and from 1% to 30% for the CAV compared to the baseline.

Battery State-of-Health Aware Path Planning for a Mars Rover

A rover mission consists of visiting waypoints to gather scientific samples based on set requirements. However, rovers face operational uncertainties during the mission, affecting the performance of its electrical and mechanical components and overall mission success. Hence, it is critical to have a decision-making framework that is aware of the health state of the components when planning the path of the vehicle. In particular, battery degradation, and consequently the battery State of Health (SOH), can affect the optimality of decisions made by the autonomous system in the long term. This paper presents a decision-making system that incorporates information on the energy drawn from the battery (based on the velocity of the vehicle), terrain conditions, and model-based prognostic modules to assess impact on the battery state of charge (SoC). The decision-making system was formulated as a Markov Decision Process (MDP) to reach the goal destination by sending commands in a determined amount of time, while maintaining the battery SoC within the policy stated. The MDP problem was programmed using the open-source framework POMDPs.jl, which has a variety of online and offline solvers. To solve the MDP problem online, we used Monte Carlo Tree Search (MCTS). Results from simulations demonstrate the effect that battery degradation and charging plans have on decision-making.

A Conflict Resolution Strategy at a Taxiway Intersection by Combining a Monte Carlo Tree Search with Prior Knowledge

With the escalating complexity of surface operations at large airports, the conflict risk for aircraft taxiing has correspondingly increased. Usually, the Air Traffic Controllers (ATCOs) generate route, speed and holding instructions to resolve conflicts. In this paper, we introduce a conflict resolution framework that incorporates prior knowledge by integrating a Multi-Layer Perceptron (MLP) neural network into the Monte Carlo Tree Search (MCTS) approach. The neural network is trained to learn the allocation strategy for waiting time extracted from actual aircraft taxiing trajectory data. Subsequently, the action probability distribution generated with the neural network is embedded into the MCTS algorithm as a heuristic evaluation function to guide the search process in finding the optimal conflict resolution strategy. Experimental results show that the average conflict resolution rate is 96.8% in different conflict scenarios, and the taxiing time required to resolve conflicts is reduced by an average of 42.77% compared to the taxiing time in actual airport surface operations.

A review on the application of reinforcement learning to the 2048 game

In light of the rapid development of Artificial Intelligence (AI), this paper pries into the application of reinforcement learning on the 2048 game which has important implications on the algorithms ability to solve various real-life problems including but not limited to robotics, auto-driving, and financial trading. The paper summarizes the possible usage of the 2048 model in Q-learning, convolutional neural networks, genetic, and Monte Carlo tree search algorithms. Then the paper scopes into the current progress and methods used in solving the 2048 game with reinforcement learning. Starting off with Q-learning, its implementation and limitations, such as large state space, sparse rewards, limited exploration, computational complexity, and a lack of human-like intuition, are illustrated. Deep Q-learning handles the problem of large state space from Q-learning. However, most of the disadvantages that are found in Q-learning are shared by Deep Q-learning and hence should become the focus of future research on using reinforcement learning to solve 2048.

The Monte Carlo tree search based bandwidth slicing allocation algorithm

The Monte Carlo tree search based bandwidth slicing allocation algorithm

EarlGAN: An enhanced actor–critic reinforcement learning agent-driven GAN for de novo drug design

Deep generative models, such as Generative Adversarial Networks (GANs), have attracted the attention of researchers in the denovo drug design field. However, traditional GANs are typically used for image processing. Therefore, they are unsuitable for Simplified Molecular-Input Line-Entry System (SMILES) strings due to their discrete nature. Previous studies addressed this problem by combining Reinforcement Learning (RL) approaches with Monte Carlo tree search. However, for large chemical datasets, the molecule generation process is time-consuming due to the lengthy atom-by-atom sampling process with cumulative reward, an essence of the Monte Carlo tree search-based RL approaches. To address this problem, we propose an enhanced actor–critic RL agent-driven GAN, called EarlGAN, for denovo drug design. Specifically, EarlGAN’s generator acts as an actor to generate SMILES strings, and the discriminator acts as a critic to perform discrimination. EarlGAN makes autoregressive predictions at the atomic level. While the generator is based on previously generated atoms, the discriminator discriminates using a bidirectional pass over the atoms, including the current atom that is being predicted. We integrate moment, global-level discrimination rewards, and information entropy maximization. The moment rewards reduce the computation time, and the global-level rewards ensure the consistency of the molecule, whereas the information entropy maximization leads to a more diverse sample generation. Experiments and ablation studies verify the effectiveness of EarlGAN for denovo drug design on the QM9 and ZINC datasets. In addition, the visualization analysis provides insight into EarlGAN and supports our conclusion.

A Continuous Action Space Tree search for INverse desiGn (CASTING) framework for materials discovery

Material properties share an intrinsic relationship with their structural attributes, making inverse design approaches crucial for discovering new materials with desired functionalities. Reinforcement Learning (RL) approaches are emerging as powerful inverse design tools, often functioning in discrete action spaces. This constrains their application in materials design problems, which involve continuous search spaces. Here, we introduce an RL-based framework CASTING (Continuous Action Space Tree Search for inverse design), that employs a decision tree-based Monte Carlo Tree Search (MCTS) algorithm with continuous space adaptation through modified policies and sampling. Using representative examples like Silver (Ag) for metals, Carbon (C) for covalent systems, and multicomponent systems such as graphane, boron nitride, and complex correlated oxides, we showcase its accuracy, convergence speed, and scalability in materials discovery and design. Furthermore, with the inverse design of super-hard Carbon phases, we demonstrate CASTING’s utility in discovering metastable phases tailored to user-defined target properties and preferences.

Data-driven approaches for identifying hyperparameters in multi-step retrosynthesis.

The multi-step retrosynthesis problem can be solved by a search algorithm, such as Monte Carlo tree search (MCTS). The performance of multistep retrosynthesis, as measured by a trade-off in search time and route solvability, therefore depends on the hyperparameters of the search algorithm. In this paper, we demonstrated the effect of three MCTS hyperparameters (number of iterations, tree depth, and tree width) on metrics such as Linear integrated speed-accuracy score (LISAS) and Inverse efficiency score which consider both route solvability and search time. This exploration was conducted by employing three data-driven approaches, namely a systematic grid search, Bayesian optimization over an ensemble of molecules to obtain static MCTS hyperparameters, and a machine learning approach to dynamically predict optimal MCTS hyperparameters given an input target molecule. With the obtained results on the internal dataset, we demonstrated that it is possible to identify a hyperparameter set which outperforms the current AiZynthFinder default setting. It appeared optimal across a variety of target input molecules, both on proprietary and public datasets. The settings identified with the in-house dataset reached a solvability of 93 % and median search time of 151 s for the in-house dataset, and a 74 % solvability and 114 s for the ChEMBL dataset. These numbers can be compared to the current default settings which solved 85 % and 73 % during a median time of 110s and 84 s, for in-house and ChEMBL, respectively.

An Efficient Aircraft Conflict Detection and Resolution Method Based on an Improved Reinforcement Learning Framework

With the steady increase of air traffic column, an auxiliary decision tool is required to compensate the operation redundancy deficiency of more sectors of air traffic control. To solve the problem of nonconflict high-density departure and arrival traffic flow, this method is expected to rapidly establish and maintain safe separation with more flexible changing strategies for aircraft heading and speed. This paper proposes an improved reinforcement learning framework to achieve conflict detection and resolution. The proposed framework includes the first development of an air traffic flow model based on a multiagent Markov decision process. The goal reward function was then maximized by improved Monte-Carlo tree search combined with an upper confidence bound tree. Three simulation scenarios were designed for illustrating the improvements of the proposed algorithm, with the results indicating that the algorithm could establish and maintain safe separation between 20 agents in the simplified hexagon-shaped airspace of Huadong, China. Furthermore, the proposed method was demonstrated to reduce the number of conflicts between aircraft agents by up to 26.32% compared to previous research.

The importance of V2X simulation: An in-depth comparison of intersection control algorithms using a high-fidelity communication simulation

The rapid evolution of vehicle-to-vehicle and vehicle-to-infrastructure communication opens doors to various control algorithms, one particular domain being intersection control. Several researchers have proposed communication-based intersection control algorithms omitting traditional traffic lights with the promise of enhanced throughput and improved traffic safety. On the other hand, the majority of these algorithms ignore the uncertainties and delays of communication and overlook new cyberattack vectors that are opened by connected traffic. The present study employs a highly detailed simulation of vehicular communication to highlight the sensitivity of different autonomous intersection control algorithms (both centralized and decentralized) to communication-related imperfections. The paper investigates five different algorithms borrowed from the literature in a comparative way. This research focuses on traffic-related parameters such as average speed, occupancy, and network throughput while also analyzing communication-related parameters (e.g., packet loss, computational demand) and considering possible attack vectors. Simulation results suggest that even the simplest control logic is sensitive to communication failures, degrading intersection throughput below traditional intersection control or even compromising traffic safety. For centralized algorithms, in the presence of noise, average speeds drop significantly, suggesting reduced intersection throughput and even gridlock for the First Come First Serve algorithm. Decentralized algorithms can be heavily affected by incoming message orders, which can lead to dangerous situations. During the simulations, multiple collisions were registered for the Monte-Carlo Tree Search algorithm. More complex algorithms that rely on accurate prediction on vehicle trajectories are more sensitive to noise and even produce accidents in the simulation. In conclusion, regardless of control architecture, the evaluated algorithms require additional fallback solutions and redundancies to retain traffic safety.

AxOTreeS : A Tree S earch Approach to Synthesizing FPGA-based A ppro x imate O perators

Approximate computing (AxC) provides the scope for achieving disproportionate gains in a system’s power, performance, and area (PPA) metrics by leveraging an application’s inherent error-resilient behavior (BEHAV). Trading computational accuracy for performance gains makes AxC an attractive proposition for implementing computationally complex AI/ML-based applications on resource-constrained embedded systems. The growing diversity of application domains using AI/ML has also led to the increasing usage of FPGA-based embedded systems. However, implementing AxC for FPGAs has primarily been limited to the post-processing of ASIC-optimized approximate operators (AxOs). This approach usually involves selecting from a set of AxOs that have been optimized for a gate-based implementation in an ASIC. While such an approach does allow leveraging existing knowledge of ASIC-based AxO design, it limits the scope for considering the challenges and opportunities associated with FPGA’s LUT-based computation structures. Similarly, the few works considering the LUT-based computing for AxO design use generic optimization approaches that do not allow integrating problem-specific prior knowledge—empirical and/or statistical. To this end, we propose a novel tree search-based approach to AxO synthesis for FPGAs. Specifically, we present a design methodology using Monte Carlo Tree Search (MCTS)-based search tree traversal that allows the designer to integrate statistical data, such as correlation, into the AxOs optimization. With the proposed methods, we report improvements over standard MCTS algorithm-based results as well as improved hypervolume for both operator-level and application-specific DSE, compared to state-of-the-art design methodologies.

Evolving the MCTS Upper Confidence Bounds for Trees Using a Semantic-Inspired Evolutionary Algorithm in the Game of Carcassonne

Monte Carlo Tree Search (MCTS) is a sampling best-first method to search for optimal decisions. The success of MCTS depends heavily on how the tree is built and the selection process plays a fundamental role in this. One particular selection mechanism that has proved to be reliable is based on the Upper Confidence Bounds for Trees (UCT). The UCT attempts to balance exploration and exploitation by considering the values stored in the statistical tree of the MCTS. However, some tuning of the MCTS UCT is necessary for this to work well. In this work, we use Evolutionary Algorithms (EAs) to evolve mathematical expressions with the goal to substitute the UCT formula and use the evolved expressions in MCTS. More specifically, we evolve expressions by means of our proposed Semantic-inspired Evolutionary Algorithm in MCTS approach (SIEA-MCTS). This is inspired by semantics in Genetic Programming (GP), where the use of fitness cases is seen as a requirement to be adopted in GP. Fitness cases are normally used to determine the fitness of individuals and can be used to compute the semantic similarity (or dissimilarity) of individuals. However, fitness cases are not available in MCTS. We extend this notion by using multiple reward values from MCTS that allow us to determine both the fitness of an individual and its semantics. By doing so, we show how SIEA-MCTS is able to successfully evolve mathematical expressions that yield better or competitive results compared to UCT without the need of tuning these evolved expressions. We compare the performance of the proposed SIEA-MCTS against MCTS algorithms, MCTS Rapid Action Value Estimation algorithms, three variants of the *-minimax family of algorithms, a random controller and two more EA approaches. We consistently show how SIEA-MCTS outperforms most of these intelligent controllers in the challenging game of Carcassonne, whose state-space complexity is, approx., 10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{40}$</tex-math></inline-formula> .

Hybrid Reinforcement Learning for Power Transmission Network Self-Healing Considering Wind Power.

Transmission network self-healing considering uncertain wind power becomes crucial with increasing penetration of wind power. A hybrid reinforcement learning (HRL) method combining offline self-learning with online Monte Carlo tree search (MCTS) is designed to deal with the strong uncertainty induced by wind power restoration. The HRL method trains a policy network with offline self-learning based on historical wind and transmission system data. It then applies the policy network to guide MCTS to realize step-by-step transmission network self-healing based on real-time and forecast data in different wind power scenarios. Besides, a model predictive control method for active power dispatch is proposed to improve wind power generation credibility during self-healing. Simulation results of both test and real-life power systems demonstrate that the proposed method can realize online transmission system self-healing reliably. Comparisons among different reinforcement learning methods indicate that the number of scenarios dominated by HRL is more than twice that dominated by MCTS and a dozen times that dominated by deep Q-network. Meanwhile, the online method is more flexible in uncertain wind power scenarios than optimization methods.

Monte Carlo tree search-based deep reinforcement learning for flexible operation & maintenance optimization of a nuclear power plant

Nuclear power plants (NPPs) are required to operate on a flexible profitable production plan while guaranteeing high safety standards. Deep reinforcement learning (DRL) is an effective method to find the most profitable operation & maintenance (O&M) strategy to adopt in a complex system. However, profit-driven only DRL neglects safety-related issues. In this paper, we propose a DRL approach to solve single-objective sequential decision problems (SOSDPs) and multi-objective sequential decision problems (MOSDPs) to find O&M strategies that trade off reliability and profit. The combinatorial problem related with the training of the RL agent to search for the optimal solution is addressed by Monte Carlo tree search (MCTS), whose performance is compared with the traditionally adopted proximal policy optimization (PPO) & imitation learning (IL). A case study is considered for demonstration.

Explainable production planning under partial observability in high-precision manufacturing

Conceptually, high-precision manufacturing is a sequence of production and measurement steps, where both kinds of steps require to use non-deterministic models to represent production and measurement tolerances. This paper demonstrates how to effectively represent these manufacturing processes as Partially Observable Markov Decision Processes (POMDP) and derive an offline strategy with state-of-the-art Monte Carlo Tree Search (MCTS) approaches. In doing so, we face two challenges: a continuous observation space and explainability requirements from the side of the process engineers. As a result, we find that a tradeoff between the quantitative performance of the solution and its explainability is required. In a nutshell, the paper elucidates the entire process of explainable production planning: We design and validate a white-box simulation from expert knowledge, examine state-of-the-art POMDP solvers, and discuss our results from both the perspective of machine learning research and as an illustration for high-precision manufacturing practitioners.

User Cold-Start Learning In Recommender Systems Using Monte Carlo Tree Search

We consider the cold-start task for new users of a recommender system whereby a new user is asked to rate a few items with the aim of quickly discovering the user’s preferences. This is a combinatorial stochastic learning task, and so difficult in general. In this paper we study the use of Monte Carlo Tree Search (MCTS) to dynamically select the sequence of items presented to a new user. We find that the MCTS-based cold-start approach is able to consistently quickly identify the preferences of a user with significantly higher accuracy than with either a decision-tree or a state of the art bandit-based approach without incurring higher regret i.e the learning performance is fundamentally superior to that of the state of the art. This boost in recommender accuracy is achieved in a computationally lightweight fashion. The MCTS approach is flexible in the sense that it can readily extended to incorporate different types of user feedback including explicit ratings, ranked comparisons and missing not at random data.

Automatic mechanistic inference from large families of Boolean models generated by Monte Carlo tree search.

Many important processes in biology, such as signaling and gene regulation, can be described using logic models. These logic models are typically built to behaviorally emulate experimentally observed phenotypes, which are assumed to be steady states of a biological system. Most models are built by hand and therefore researchers are only able to consider one or perhaps a few potential mechanisms. We present a method to automatically synthesize Boolean logic models with a specified set of steady states. Our method, called MC-Boomer, is based on Monte Carlo Tree Search an efficient, parallel search method using reinforcement learning. Our approach enables users to constrain the model search space using prior knowledge or biochemical interaction databases, thus leading to generation of biologically plausible mechanistic hypotheses. Our approach can generate very large numbers of data-consistent models. To help develop mechanistic insight from these models, we developed analytical tools for multi-model inference and model selection. These tools reveal the key sets of interactions that govern the behavior of the models. We demonstrate that MC-Boomer works well at reconstructing randomly generated models. Then, using single time point measurements and reasonable biological constraints, our method generates hundreds of thousands of candidate models that match experimentally validated in-vivo behaviors of the Drosophila segment polarity network. Finally we outline how our multi-model analysis procedures elucidate potentially novel biological mechanisms and provide opportunities for model-driven experimental validation.

Dynamic Data-Driven Application System for Flow Field Prediction with Autonomous Marine Vehicles

Efficiently predicting high-resolution and accurate flow fields through networked autonomous marine vehicles (AMVs) is crucial for diverse applications. Nonetheless, a research gap exists in the seamless integration of data-driven flow modeling, real-time data assimilation from flow sensing, and the optimization of AMVs’ sensing strategies, culminating in a closed-loop dynamic data-driven application system (DDDAS). This article presents a novel DDDAS that systematically integrates flow modeling, data assimilation, and adaptive flow sensing using networked AMVs. It features a hybrid data-driven flow model, uniting a neural network for trend prediction and a Gaussian process model for residual fitting. The neural network architecture is designed using knowledge extracted from historic flow data through tidal harmonic analysis, enhancing its capability in flow prediction. The Kriged ensemble transform Kalman filter is introduced to assimilate spatially correlated flow-sensing data from AMVs, enabling effective model learning and accurate spatiotemporal flow prediction, while forming the basis for optimizing AMVs’ flow-sensing paths. A receding horizon strategy is proposed to implement non-myopic optimal path planning, and a distributed strategy of implementing Monte Carlo tree search is proposed to solve the resulting large-scale tree searching-based optimization problem. Computer simulations, employing underwater gliders as sensing networks, demonstrate the effectiveness of the proposed DDDAS in predicting depth-averaged flow in nearshore ocean environments.

Adaptive Discretization using Voronoi Trees for Continuous POMDPs

Solving continuous Partially Observable Markov Decision Processes (POMDPs) is challenging, particularly for high-dimensional continuous action spaces. To alleviate this difficulty, we propose a new sampling-based online POMDP solver, called A daptive D iscretization using V oronoi T rees (ADVT). It uses Monte Carlo Tree Search in combination with an adaptive discretization of the action space as well as optimistic optimization to efficiently sample high-dimensional continuous action spaces and compute the best action to perform. Specifically, we adaptively discretize the action space for each sampled belief using a hierarchical partition called Voronoi tree, which is a Binary Space Partitioning that implicitly maintains the partition of a cell as the Voronoi diagram of two points sampled from the cell. ADVT uses the estimated diameters of the cells to form an upper-confidence bound on the action value function within the cell, guiding the Monte Carlo Tree Search expansion and further discretization of the action space. This enables ADVT to better exploit local information with respect to the action value function, allowing faster identification of the most promising regions in the action space, compared to existing solvers. Voronoi trees keep the cost of partitioning and estimating the diameter of each cell low, even in high-dimensional spaces where many sampled points are required to cover the space well. ADVT additionally handles continuous observation spaces, by adopting an observation progressive widening strategy, along with a weighted particle representation of beliefs. Experimental results indicate that ADVT scales substantially better to high-dimensional continuous action spaces, compared to state-of-the-art methods.