Monte Carlo Tree Research Articles

Monte Carlo tree search applied to design of wireless power transfer system

Monte Carlo tree search applied to design of wireless power transfer system

A crossword solving system based on Monte Carlo tree search

Although the development of AI in games is remarkable, intelligent machines still lag behind humans in games that require the ability of language understanding. In this paper, we focus on the crossword puzzle resolution task. Solving crossword puzzles is a challenging task since it requires the ability to answer natural language questions with knowledge and the ability to execute a search over possible answers to find an optimal set of solutions for the grid. Previous solutions are devoted to exploiting heuristic strategies in search to find solutions while having limited ability to explore the search space. We build a comprehensive system for crossword puzzle resolution based on Monte Carlo Tree Search (MCTS). As far as we know, we are the first to model the crossword puzzle resolution problem as a Markov Decision Process and apply the MCTS to solve it. We construct a dataset for crossword puzzle resolution based on daily puzzles from The New York Times with detailed specifications of both the puzzle and clue database selection. Our method achieves state-of-the-art performance on the dataset. The code of the system and experiments in this paper is publicly available: https://www.github.com/lhlclhl/CP.

Monte Carlo tree search for dynamic shortest‐path interdiction

AbstractWe present a reinforcement learning‐based heuristic for a two‐player interdiction game called the dynamic shortest path interdiction problem (DSPI). The DSPI involves an evader and an interdictor who take turns in the problem, with the interdictor selecting a set of arcs to attack and the evader choosing an arc to traverse at each step of the game. Our model employs the Monte Carlo tree search framework to learn a policy for the players using randomized roll‐outs. This policy is stored as an asymmetric game tree and can be further refined as the game unfolds. We leverage alpha–beta pruning and existing bounding schemes in the literature to prune suboptimal branches. Our numerical experiments demonstrate that the prescribed approach yields near‐optimal solutions in many cases and allows for flexibility in balancing solution quality and computational effort.

Domain independent heuristics for online stochastic contingent planning

AbstractPartially observable Markov decision processes (POMDP) are a useful model for decision-making under partial observability and stochastic actions. Partially Observable Monte-Carlo Planning (POMCP) is an online algorithm for deciding on the next action to perform, using a Monte-Carlo tree search approach, based on the UCT algorithm for fully observable Markov-decision processes. POMCP develops an action-observation tree, and at the leaves, uses a rollout policy to provide a value estimate for the leaf. As such, POMCP is highly dependent on the rollout policy to compute good estimates, and hence identify good actions. Thus, many practitioners who use POMCP are required to create strong, domain-specific heuristics. In this paper, we model POMDPs as stochastic contingent planning problems. This allows us to leverage domain-independent heuristics that were developed in the planning community. We suggest two heuristics, the first is based on the well-known $$h_{add}$$ h add heuristic from classical planning, and the second is computed in belief space, taking the value of information into account.

Model inductive bias enhanced deep reinforcement learning for robot navigation in crowded environments

Navigating mobile robots in crowded environments poses a significant challenge and is essential for the coexistence of robots and humans in future intelligent societies. As a pragmatic data-driven approach, deep reinforcement learning (DRL) holds promise for addressing this challenge. However, current DRL-based navigation methods have possible improvements in understanding agent interactions, feedback mechanism design, and decision foresight in dynamic environments. This paper introduces the model inductive bias enhanced deep reinforcement learning (MIBE-DRL) method, drawing inspiration from a fusion of data-driven and model-driven techniques. MIBE-DRL extensively incorporates model inductive bias into the deep reinforcement learning framework, enhancing the efficiency and safety of robot navigation. The proposed approach entails a multi-interaction network featuring three modules designed to comprehensively understand potential agent interactions in dynamic environments. The pedestrian interaction module can model interactions among humans, while the temporal and spatial interaction modules consider agent interactions in both temporal and spatial dimensions. Additionally, the paper constructs a reward system that fully accounts for the robot’s direction and position factors. This system's directional and positional reward functions are built based on artificial potential fields (APF) and navigation rules, respectively, which can provide reasoned evaluations for the robot's motion direction and position during training, enabling it to receive comprehensive feedback. Furthermore, the incorporation of Monte-Carlo tree search (MCTS) facilitates the development of a foresighted action strategy, enabling robots to execute actions with long-term planning considerations. Experimental results demonstrate that integrating model inductive bias significantly enhances the navigation performance of MIBE-DRL. Compared to state-of-the-art methods, MIBE-DRL achieves the highest success rate in crowded environments and demonstrates advantages in navigation time and maintaining a safe social distance from humans.

PPB-MCTS: A novel distributed-memory parallel partial-backpropagation Monte Carlo tree search algorithm

Monte-Carlo Tree Search (MCTS) is an adaptive and heuristic tree-search algorithm designed to uncover sub-optimal actions at each decision-making point. This method progressively constructs a search tree by gathering samples throughout its execution. Predominantly applied within the realm of gaming, MCTS has exhibited exceptional achievements. Additionally, it has displayed promising outcomes when employed to solve NP-hard combinatorial optimization problems. MCTS has been adapted for distributed-memory parallel platforms. The primary challenges associated with distributed-memory parallel MCTS are the substantial communication overhead and the necessity to balance the computational load among various processes. In this work, we introduce a novel distributed-memory parallel MCTS algorithm with partial backpropagations, referred to as Parallel Partial-Backpropagation MCTS (PPB-MCTS). Our design approach aims to significantly reduce the communication overhead while maintaining, or even slightly improving, the performance in the context of combinatorial optimization problems. To address the communication overhead challenge, we propose a strategy involving transmitting an additional backpropagation message. This strategy avoids attaching an information table to the communication messages exchanged by the processes, thus reducing the communication overhead. Furthermore, this approach contributes to enhancing the decision-making accuracy during the selection phase. The load balancing issue is also effectively addressed by implementing a shared transposition table among the parallel processes. Furthermore, we introduce two primary methods for managing duplicate states within distributed-memory parallel MCTS, drawing upon techniques utilized in addressing duplicate states within sequential MCTS. Duplicate states can transform the conventional search tree into a Directed Acyclic Graph (DAG). To evaluate the performance of our proposed parallel algorithm, we conduct an extensive series of experiments on solving instances of the Job-Shop Scheduling Problem (JSSP) and the Weighted Set-Cover Problem (WSCP). These problems are recognized for their complexity and classified as NP-hard combinatorial optimization problems with considerable relevance within industrial applications. The experiments are performed on a cluster of computers with many cores. The empirical results highlight the enhanced scalability of our algorithm compared to that of the existing distributed-memory parallel MCTS algorithms. As the number of processes increases, our algorithm demonstrates increased rollout efficiency while maintaining an improved load balance across processes.

Tackling rolling security-constrained unit commitment under uncertainty with Monte Carlo value search and empirical knowledge

Recently, the growing uncertainty in the power system has created an increasing need for security-constrained unit commitment (SCUC) solved on a rolling perspective (i.e., RSCUC). This puts a high requirement on the decision efficiency of the solution algorithm. To overcome the challenge, this paper proposes a POML-MCVS algorithm, which integrates a probability-oriented machine learning (POML) technique with Monte-Carlo value search (MCVS) algorithm to improve overall performance. First, POML is employed to identify the decision probabilities of units through leveraging empirical knowledge, and determine the on/off states of units with high decision probabilities. This facilitates improving the decision efficiency while maintaining the decision accuracy as much as possible. Then, MCVS is employed to optimize the rest variables and make the rolling decisions. As a modified version of Monte Carlo tree search, MCVS is able to detect the promising decisions, and evaluate their values efficiently through constructing the search tree asymmetrically, which further improves the decision efficiency. Numerical simulations conducted on a modified IEEE 39-bus system and a real 2778-bus system demonstrate the effectiveness of the proposed algorithm.

Stateful black-box fuzzing for encryption protocols and its application in IPsec

Owing to the rapid development of information security technology, the security analysis of encryption protocols has received widespread attention. In this paper, we propose a stateful black-box encryption protocol fuzzing method to analyze the security of real-world black-box encryption protocol programs and devices. This method does not rely on the source code but uses captured packets as input, performs state selection based on a Monte Carlo tree search algorithm, and processes the encryption/decryption conversion of mutant test cases based on the intermediate mapper. It sends test cases and collects responses in interactive communication with the tested program and dynamically optimizes the corpus based on the collected state information. Based on this method, we develop SBEPFuzz and primarily analyze IPsec. We evaluate SBEPFuzz on six widely used IPsec implementations. The experimental results show that SBEPFuzz achieves higher code coverage, and can discover more protocol state sequences and vulnerabilities. Furthermore, we discover four anomalies, including malformed packets triggering service crashes and abnormal interactions leading to plaintext ID leakage, which also reflect the differences in details among different implementations.

Online model adaptation in Monte Carlo tree search planning

AbstractWe propose a model-based reinforcement learning method using Monte Carlo Tree Search planning. The approach assumes a black-box approximated model of the environment developed by an expert using any kind of modeling framework and it improves the model as new information from the environment is collected. This is crucial in real-world applications, since having a complete knowledge of complex environments is impractical. The expert’s model is first translated into a neural network and then it is updated periodically using data, i.e., state-action-next-state triplets, collected from the real environment. We propose three different methods to integrate data acquired from the environment with prior knowledge provided by the expert and we evaluate our approach on a domain concerning air quality and thermal comfort control in smart buildings. We compare the three proposed versions with standard Monte Carlo Tree Search planning using the expert’s model (without adaptation), Proximal Policy Optimization (a popular model-free DRL approach) and Stochastic Lower Bounds Optimization (a popular model-based DRL approach). Results show that our approach achieves the best results, outperforming all analyzed competitors.

Comparing search algorithms on the retrosynthesis problem.

In this article we try different algorithms, namely Nested Monte Carlo Search and Greedy Best First Search, on AstraZeneca's open source retrosynthetic tool : AiZynthFinder. We compare these algorithms to AiZynthFinder's base Monte Carlo Tree Search on a benchmark selected from the PubChem database and by Bayer's chemists. We show that both Nested Monte Carlo Search and Greedy Best First Search outperform AstraZeneca's Monte Carlo Tree Search, with a slight advantage for Nested Monte Carlo Search while experimenting on a playout heuristic. We also show how the search algorithms are bounded by the quality of the policy network, in order to improve our results the next step is to improve the policy network.

Real-time stochastic flexible flow shop scheduling in a credit factory with model-based reinforcement learning

There has been a significant increase in consumer credit worldwide in recent years. The scheduling of jobs in credit factories is essential for speeding up the loan application process, which can improve the efficiency of credit factories. In this study, we propose a reinforcement learning approach for addressing the scheduling problem in credit factories, which is a stochastic flexible flow shop scheduling problem (SFFSP). First, we propose a mathematical model for the credit factory stochastic flexible flow shop scheduling problem, which abstracts the decision-making process as a semi-Markov process. Then, a reinforcement learning reward mechanism is designed based on the proposed mathematical model. After that, a self-attention neural network is used to extract state information from global and local multidimensional data, enabling each decision to consider the state of the entire process and make a decision that aligns with the global goal. Meanwhile, Monte Carlo Tree Search (MCTS) is utilised to enhance the training effect and sample utilisation of reinforcement learning. Finally, we conduct extensive experiments and demonstrate that our method achieves better performance for SFFSP in credit factories compared to other approaches.

FOCUS on NOD2: Advancing IBD Drug Discovery with a User-Informed Machine Learning Framework.

In this study, we introduce the Framework for Optimized Customizable User-Informed Synthesis (FOCUS), a generative machine learning model tailored for drug discovery. FOCUS integrates domain expertise and uses Proximal Policy Optimization (PPO) to guide Monte Carlo Tree Search (MCTS) to efficiently explore chemical space. It generates SMILES representations of potential drug candidates, optimizing for druggability and binding efficacy to NOD2, PEP, and MCT1 receptors. The model is highly interpretive, allowing for user-feedback and expert-driven adjustments based on detailed cycle reports. Employing tools like SHAP and LIME, FOCUS provides a transparent analysis of decision-making processes, emphasizing features such as docking scores and interaction fingerprints. Comparative studies with Muramyl Dipeptide (MDP) demonstrate improved interaction profiles. FOCUS merges advanced machine learning with expert insight, accelerating the drug discovery pipeline.

Lookahead Pathology in Monte-Carlo Tree Search

Monte-Carlo Tree Search (MCTS) is a search paradigm that first found prominence with its success in the domain of computer Go. Early theoretical work established the soundness and convergence bounds for Upper Confidence bounds applied to Trees (UCT), the most popular instantiation of MCTS; however, there remain notable gaps in our understanding of how UCT behaves in practice. In this work, we address one such gap by considering the question of whether UCT can exhibit lookahead pathology in adversarial settings --- a paradoxical phenomenon first observed in Minimax search where greater search effort leads to worse decision-making. We introduce a novel family of synthetic games that offer rich modeling possibilities while remaining amenable to mathematical analysis. Our theoretical and experimental results suggest that UCT is indeed susceptible to pathological behavior in a range of games drawn from this family.

Autonomous Maneuver Planning for Small-Body Reconnaissance via Reinforcement Learning

This paper presents a reinforcement learning (RL) based approach for autonomous maneuver planning of low-altitude flybys for site-specific reconnaissance of small bodies. Combined with Monte Carlo tree search and deep neural networks, the proposed method generates optimal maneuvers, even under complex dynamics and abstractly science goals. Formulating the mission objective as an observability function, the RL issue can be framed in terms of a Markov decision process. The neural network, trained by a novel policy gradient algorithm with a clipped surrogate objective, learns both policy and value functions that map the action and state spaces to the expected long-term return. An adaptive refinement search technique is applied to further enhance the trained policy network, finding optimal maneuvers from the policy distributions. Experiment results on a simulated reconnaissance mission around asteroid Itokawa illustrate the efficiency and robustness of the proposed approach in achieving multitarget observation.

MoLPC2: improved prediction of large protein complex structures and stoichiometry using Monte Carlo Tree Search and AlphaFold2.

Today, the prediction of structures of large protein complexes solely from their sequence information requires prior knowledge of the stoichiometry of the complex. To address this challenge, we have enhanced the Monte Carlo Tree Search algorithms in MoLPC to enable the assembly of protein complexes while simultaneously predicting their stoichiometry. In MoLPC2, we have improved the predictions by allowing sampling alternative AlphaFold predictions. Using MoLPC2, we accurately predicted the structures of 50 out of 175 nonredundant protein complexes (TM-score ≥ 0.8) without knowing the stoichiometry. MoLPC2 provides new opportunities for predicting protein complex structures without stoichiometry information. MoLPC2 is freely available at https://github.com/hychim/molpc2. A notebook is also available from the repository for easy use.

Optimizing multi-objective design, planning, and operation for sustainable energy sharing districts considering electrochemical battery longevity

The urgent need to address the global warming crisis has led to a paradigm shift in energy production from traditional fossil fuels to renewable sources such as geothermal, solar, and wind. This transition necessitates efficient energy management strategies to enhance renewable energy utilization, reduce microgrid dependency, and establish a more stable power grid. This study explores the novel integration of interactive energy sharing networks utilizing electrochemical battery storage, emphasizing detailed modeling of battery degradation, smart energy management, and multi-criteria decision-making. This research involves an innovative method combining Monte Carlo Tree Search technique and a multi-criteria approach for energy management in district communities. The research delves into the complexities of integrated multi-energy systems, considering structural designs, advanced modeling methods, and multi-criteria predictions. It also employs powerful machine learning methods to predict battery depreciation, demonstrating their superiority over standard semi-empirical models. The study also proposes deterministic and stochastic methods for intelligent energy management, considering unpredictable factors like weather and energy demand profiles. The research incorporates a comprehensive examination of interactive energy sharing networks, encompassing renewable-to-vehicle and vehicle-to-building interactions. Through an in-depth analysis this research highlights the advantages of multi-agent systems, time-of-use energy shifting, demand profile flattening, and micro-grid resilience.

The application of chessboard game based on integrated learning and UCT algorithm in mental health and emotional regulation

Go plays a very prominent role in quality education and mental health education. Go can cultivate character, make friends, improve interpersonal relationships, and promote mental health. To apply it to mental health and emotional regulation, the AlphaGo series algorithms are analyzed. To address the unstable deep reinforcement learning training, low training data utilization, model sensitivity to hyper-parameters, and heavy parameter tuning burden, the algorithm is optimized by combining ensemble learning and advanced deep reinforcement learning. Then, the weighted voting method is applied to the selection step of sub nodes in the Monte Carlo tree search process. At the same time, asynchronous parallelization improvements are implemented to improve the efficiency of the Monte Carlo tree search algorithm. Finally, a board game based on ensemble learning and improved Monte Carlo tree search algorithm is obtained. Through experimental analysis, the average winning rate of the system was 93.36 %. The number of users who were very satisfied with the game was 83.96 %. After intervention, the depression and anxiety scores of the experimental group students were significantly lower than the control group (P < 0.05). The designed game can effectively alleviate user anxiety and depression, providing new ideas for the psychotherapy.

Optimizing Camera Motion with MCTS and Target Motion Modeling in Multi-Target Active Object Tracking

In this work, we are dedicated to multi-target active object tracking (AOT), where the goal is to achieve continuous tracking of targets through real-time control of camera. This form of active camera control can be applied to unmanned aerial vehicles (UAV), intelligent robots, and sports events. Our work is conducted in an environment featuring multiple cameras and targets, where our goal is to maximize target coverage. Contrasting with previous research, our work introduces additional degrees of freedom for the cameras, allowing them not only to rotate but also to move along boundary lines. In addition, we model the motion of target to predict the future position of the target in environment. With target’s future position, we use Monte Carlo Tree Search (MCTS) method to find the optimal action of camera. Since the action space is large, we propose to leverage the action selection from multi-agent reinforcement learning (MARL) network to prune the search tree of Monte Carlo Tree Search method, so as to find the optimal action more efficiently. We establish a multi-target 2D environment to simulate several sports games, and experimental results demonstrate that our method can effectively improve the target coverage. The code is available at: http://github.com/HopeChanger/ActiveObjectTracking .

Retrosynthesis Zero: Self-Improving Global Synthesis Planning Using Reinforcement Learning.

The field of computer-aided synthesis planning (CASP) has witnessed significant growth in recent years. Still, many CASP programs rely on large data sets to train neural networks, resulting in limitations due to the data quality and prior knowledge from chemists. In response, we propose Retrosynthesis Zero (ReSynZ), a reaction template-based method that combines Monte Carlo Tree Search with reinforcement learning inspired by AlphaGo Zero. Unlike other single-step reaction template-based CASP methods, ReSynZ takes complete synthesis paths for complex molecules, determined by reaction rules, as input for training the neural network. ReSynZ enables neural networks trained with relatively small reaction data sets (tens of thousands of data) to generate multiple synthesis pathways for a target molecule and suggest possible reaction conditions. On multiple data sets of molecular retrosynthesis, ReSynZ demonstrates excellent predictive performance compared to existing algorithms. The advantages, such as self-improving model features, flexible reward settings, the potential to surpass human limitations in chemical synthesis route planning, and others, make ReSynZ a valuable tool in chemical synthesis design.

Enhancing Autonomous Underwater Vehicle Decision Making through Intelligent Task Planning and Behavior Tree Optimization

The expansion of underwater scenarios and missions highlights the crucial need for autonomous underwater vehicles (AUVs) to make informed decisions. Therefore, developing an efficient decision-making framework is vital to enhance productivity in executing complex tasks within tight time constraints. This paper delves into task planning and reconstruction within the AUV control decision system to enable intelligent completion of intricate underwater tasks. Behavior trees (BTs) offer a structured approach to organizing the switching structure of a hybrid dynamical system (HDS), originally introduced in the computer game programming community. In this research, an intelligent search algorithm, MCTS-QPSO (Monte Carlo tree search and quantum particle swarm optimization), is proposed to bolster the AUV’s capacity in planning complex task decision control systems. This algorithm tackles the issue of the time-consuming manual design of control systems by effectively integrating BTs. By assessing a predefined set of subtasks and actions in tandem with the complex task scenario, a reward function is formulated for MCTS to pinpoint the optimal subtree set. The QPSO algorithm is then leveraged for subtree integration, treating it as an optimal path search problem from the root node to the leaf node. This process optimizes the search subtree, thereby enhancing the robustness and security of the control architecture. To expedite search speed and algorithm convergence, this paper recommends reducing the search space by pre-grouping conditions and states within the behavior tree. The efficacy and superiority of the proposed algorithm are validated through security and timeliness evaluations of the BT, along with comparisons with other algorithms for automatic AUV decision control behavior tree design. Ultimately, the effectiveness and superiority of the proposed algorithm are corroborated through simulations on a multi-AUV complex task platform, showcasing its practical applicability and efficiency in real-world underwater scenarios.