Monte Carlo Tree Search Algorithm Research Articles

Automating Aircraft Scanning for Inspection or 3D Model Creation with a UAV and Optimal Path Planning

Visual inspections of aircraft exterior surfaces are required in aircraft maintenance routines for identifying possible defects such as dents, cracks, leaking, broken or missing parts, etc. This process is time-consuming and is also prone to error if performed manually. Therefore, it has become a trend to use mobile robots equipped with visual sensors to perform automated inspections. For such a robotic inspection, a digital model of the aircraft is usually required for planning the robot’s path, but a CAD model of the entire aircraft is usually inaccessible to most maintenance shops. It is very labor-intensive and time-consuming to generate an accurate digital model of an aircraft, or even a large portion of it, because the scanning work still must be performed manually or by a manually controlled robotic system. This paper presents a two-stage approach of automating aircraft scanning with an unmanned aerial vehicle (UAV) or autonomous drone equipped with a red–green–blue and depth (RGB-D) camera for detailed inspection or for reconstructing a digital replica of the aircraft when its original CAD model is unavailable. In the first stage, the UAV–camera system follows a predefined path far from the aircraft surface (for safety) to quickly scan the aircraft and generate a coarse model of the aircraft. Then, an optimal scanning path (much closer to the surface) in the sense of the shortest flying distance for full coverage is computed based on the coarse model. In the second stage, the UAV–camera system follows the computed path to closely inspect the surface for possible defects or scan the surface for generating a dense and precise model of the aircraft. We solved the coverage path planning (CPP) problem for the aircraft inspection or scanning using a Monte Carlo tree search (MCTS) algorithm. We also implemented the max–min ant system (MMAS) strategy to demonstrate the effectiveness of our approach. We carried out a digital experiment and the results showed that our approach can scan 70% of the aircraft surface within one hour, which is much more efficient than manual scanning.

Designing an Automatic Agent for Repeated Language–based Persuasion Games

AbstractPersuasion games are fundamental in economics and AI research and serve as the basis for important applications. However, work on this setup assumes communication with stylized messages that do not consist of rich human language. In this paper we consider a repeated sender (expert) – receiver (decision maker) game, where the sender is fully informed about the state of the world and aims to persuade the receiver to accept a deal by sending one of several possible natural language reviews. We design an automatic expert that plays this repeated game, aiming to achieve the maximal payoff. Our expert is implemented within the Monte Carlo Tree Search (MCTS) algorithm, with deep learning models that exploit behavioral and linguistic signals in order to predict the next action of the decision maker, and the future payoff of the expert given the state of the game and a candidate review. We demonstrate the superiority of our expert over strong baselines and its adaptability to different decision makers and potential proposed deals.1

A new approach for generator startup sequence online decision making with a heuristic search algorithm and graph theory

Most system restoration strategies following blackouts are obtained by offline methods, which are efficient in theory. But this may not be the case. A new method is proposed to decide the generator startup sequence during the system restoration. The Monte Carlo tree search algorithm is used to search for the generator to be recovered according to the real-time situation of the power system. All paths from the restored system to the chosen generator are obtained by Depth-First Search. And the K-shortest paths are obtained by Yen’s Algorithm to decide the best restoration path. Then, key constraints are verified. Especially, the constraint on the hot start of generators is considered. Finally, the feasible scheme with the maximum power generation is selected for implementation. In addition, the cranking process of G33 to G32 in the IEEE39-bus test system is taken as an example to illustrate the efficiency of this method. Depth-First Search detects 42 restoration paths, and Yen’s Algorithm selects three most reasonable paths from them, which greatly improves the efficiency of online decision making.

Optimizing quantum annealing schedules with Monte Carlo tree search enhanced with neural networks

Quantum annealing is a practical approach to approximately implement the adiabatic quantum computational model in a real-world setting. The goal of an adiabatic algorithm is to prepare the ground state of a problem-encoded Hamiltonian at the end of an annealing path. This is typically achieved by driving the dynamical evolution of a quantum system slowly to enforce adiabaticity. Properly optimized annealing schedules often considerably accelerate the computational process. Inspired by the recent success of deep reinforcement learning such as DeepMind’s AlphaZero, we propose a Monte Carlo tree search (MCTS) algorithm and its enhanced version boosted by neural networks—which we name QuantumZero (QZero)—to automate the design of annealing schedules in a hybrid quantum–classical framework. Both the MCTS and QZero algorithms perform remarkably well in discovering effective annealing schedules even when the annealing time is short for the 3-SAT examples considered in this study. Furthermore, the flexibility of neural networks allows us to apply transfer-learning techniques to boost QZero’s performance. We demonstrate in benchmark studies that MCTS and QZero perform more efficiently than other reinforcement learning algorithms in designing annealing schedules.

Dynamic Pricing for Charging of EVs with Monte Carlo Tree Search

As electric vehicles (EVs) are slowly becoming a common occurrence on roads, commercial EV charging is becoming a standard commercial service. With this development, charging station operators are looking for ways to make their charging services more profitable or allocate the available resources optimally. Dynamic pricing is a proven technique to increase revenue in markets with heterogeneous demand. This paper proposes a Markov Decision Process (MDP)-based approach to revenue- or utilization- maximizing dynamic pricing for charging station operators. We implement the method using a Monte Carlo Tree Search (MCTS) algorithm and evaluate it in simulation using a range of problem instances based on a real-world dataset of EV charging sessions. We show that our approach provides near-optimal pricing decisions in milliseconds for large-scale problems, significantly increasing revenue or utilization over the flat-rate baseline under a range of parameters.

A Tactical Conflict Detection and Resolution Method for En Route Conflicts in Trajectory-Based Operations

In trajectory-based operation (TBO), the four-dimensional trajectories (4DTs) of aircraft are shared with all flight-related stakeholders, which makes flights visible and controllable and helps flights arrive at a location within a fixed time range. With the technical support from TBO, the operation of flights in a route sector will be more efficient and the air traffic volume will increase. However, more flights will also result in more flight conflicts. In this study, a deterministic conflict detection and resolution (CDR) module is established to assist air traffic controllers in detecting and resolving high-density conflicts rapidly and in advance. In the conflict detection (CD) submodule, a spatial data structure with low time complexity, the R tree algorithm, is used. R tree can effectively reduce the comparison number between the 4DTs of all aircraft. The experiment results show that the computing time of the R tree presents a logarithmic curve with the increase in the number of aircraft and the efficiency of the CD is more significantly improved. In the conflict resolution (CR) submodule, considering the aircraft performance and terrain constraints, the Monte Carlo tree search (MCTS) algorithm is proposed to solve the problem of huge search space and to quickly provide an effective resolution policy for pairwise conflict. The simulation results in dense airspace indicate that the MCTS-based CR algorithm has good performance in terms of safety and efficiency.

Multi-reward Reinforcement Learning Based Bond-Order Potential to Study Strain-Assisted Phase Transitions in Phosphorene.

We introduce a multi-reward reinforcement learning (RL) approach to train a flexible bond-order potential (BOP) for 2D phosphorene based on ab initio training data sets. Our approach is based on a continuous action space Monte Carlo tree search algorithm that is general and scalable and presents an efficient multiobjective optimization scheme for high-dimensional materials design problems. As a proof-of-concept, we deploy this scheme to parametrize multiple structural and dynamical properties of 2D phosphorene polymorphs. Our RL-trained BOP model adequately captures the structure, energetics, transformation barriers, equation of state, elastic constants, and phonon dispersions of various 2D P polymorphs. We use this model to probe the impact of temperature and strain rate on the phase transition from black (α-P) to blue phosphorene (β-P) through molecular dynamics simulations. A decrease in critical strain for this phase transition with increase in temperature is observed, and the underlying atomistic mechanisms are discussed.

Neural‐augmented two‐stage Monte Carlo tree search with over‐sampling for protein folding in HP Model

This paper proposes a novel Monte Carlo tree search (MCTS) algorithm to solve the protein folding problem in HP model. There are two main challenges. First, the problem is proved to be NP‐complete. The solution space is large and it is hard to find a good solution via a search algorithm without prior knowledge of the HP model. We tackle this challenge by formulating the problem as a deterministic Markov decision process (MDP) and solve it in an AlphaZero's manner. The difference is that we design a MCTS algorithm with two stages: neural exploitation stage and random exploration stage. In the first stage, the search algorithm utilizes the knowledge from previous experience by evaluating the states with a trained neural network, while in the second stage, the states are evaluated by fast and random rollouts. It effectively reduces the number of neural inferences and computational cost. The second challenge is that the evaluation of typical MCTS cannot preserve the correct preference over the actions in our task. To address this challenge, we propose an over‐sampling mechanism that encourages the agent to search more on those actions with high rollout values. The proposed method is tested and compared in a series of experiments. Experimental results have confirmed the effectiveness of the proposed method empirically. Besides, we also visualize the obtained the best conformations and verify the proposed technical designs through an ablation study. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Monte Carlo Tree Search algorithm for the receiving containers intelligently problem among container shipping terminals

With the development of economic globalization, shipping exchanges between countries are rapidly increasing and the container throughput of shipping ports has increased quickly, and the operation of most seaport terminals in China has reached a bottleneck which puts forward new requirements for the efficiency of container shipping terminal operations. Therefore, locating for export containers intelligently is of great significance for the development of terminals. This paper focuses on receiving containers intelligently, and establishes the export container locating model based on the principle of actual shipping terminal operations. The Monte Carlo Tree Search algorithm for export container locating problem was proposed and constructed. After examples, the algorithm can effectively solve the problem that meet the constraints, which further proves the practicability of the algorithm and the correctness of the model. The research shows that the method of locating for export container based on Monte Carlo Tree Search algorithm can effectively solve the problem and maintain the green, energy-saving and sustainable development of the shipping terminal. The ideas and methods have certain academic value and reference significance for other NP-Hard problems.

A Monte-Carlo tree search algorithm for the flexible job-shop scheduling in manufacturing systems

A Monte-Carlo tree search algorithm for the flexible job-shop scheduling in manufacturing systems

UAV path planning and collision avoidance in 3D environments based on POMPD and improved grey wolf optimizer

Due to the complexity and uncertain factors of the environment, a 3D path planning algorithm is urgently needed. This paper presents a 3D optimal feasible flight path generation and collision avoidance algorithms based on partially observable Markov decision process (POMDP) and improved grey wolf optimizer (GWO) for an unmanned aerial vehicle (UAV). Firstly, a novel algorithm based on the GWO is proposed to deal with constrained optimization problem (COP) and utilized to plan a flyable path. The designed variant is called improved GWO with level comparison (GWOLC), which combines the communication mechanism and the ε-level comparison method at the same time. Secondly, aircraft collision avoidance is modeled as a Partially Observable Markov Decision Process (POMDP) and the Monte-Carlo tree search (MCTS) algorithm is used to solve it. We introduce a novel algorithm, Information Particle Filter Tree (IPFT), to solve the problem of belief update in continuous domain. Thirdly, simulation experiments are conducted in 3D environment, and numerical results showed the proposed algorithm offers good performance as measured by effectiveness, robustness, convergence, and constraint handling capabilities.

MetaGlyph: Automatic Generation of Metaphoric Glyph-based Visualization.

Glyph-based visualization achieves an impressive graphic design when associated with comprehensive visual metaphors, which help audiences effectively grasp the conveyed information through revealing data semantics. However, creating such metaphoric glyph-based visualization (MGV) is not an easy task, as it requires not only a deep understanding of data but also professional design skills. This paper proposes MetaGlyph, an automatic system for generating MGVs from a spreadsheet. To develop MetaGlyph, we first conduct a qualitative analysis to understand the design of current MGVs from the perspectives of metaphor embodiment and glyph design. Based on the results, we introduce a novel framework for generating MGVs by metaphoric image selection and an MGV construction. Specifically, MetaGlyph automatically selects metaphors with corresponding images from online resources based on the input data semantics. We then integrate a Monte Carlo tree search algorithm that explores the design of an MGV by associating visual elements with data dimensions given the data importance, semantic relevance, and glyph non-overlap. The system also provides editing feedback that allows users to customize the MGVs according to their design preferences. We demonstrate the use of MetaGlyph through a set of examples, one usage scenario, and validate its effectiveness through a series of expert interviews.

Design and optimal energy management of community microgrids with flexible renewable energy sources

Energy communities is a new, but already successful prosumer model of the local energy systems' construction. It is based on distributed energy sources and the electricity consumers’ flexibility who are the members of the community. In search of the most effective ways to interact within themselves and with the external energy system, local energy communities become platforms for exciting experiments in the field of new energy practices including local markets for flexibility, building cooperative microgrids, achieving energy autonomy, and many others. This work aims to present a unified approach to building and optimally managing the community microgrids with an internal market, given the social, environmental, and economic benefits of a particular location of such a community. A new modeling framework is introduced, based on bilevel programming and reinforcement learning, for structuring and solving the internal local market of a community microgrids, composed of entities that may exchange energy and services among themselves. The overall framework is formulated in the form of a bilevel model, where the lower level problem clears the market, while the upper level problem plays the role of the community microgrid operator (Community EMS). We strengthen the traditional bilevel problem statement by the local energy management system (Local EMS) introduction based on Monte-Carlo tree search algorithm. Our approach makes it possible to enable interaction of the local control systems for microgrids with the community microgrid operator as part bilevel programming problem solution. Numerical results obtained on the real test case of the microgrid community for the settlements located in the Transbaikal National Park (Russia), which include various renewable energy sources (wind, solar power, biomass gasifiers) and storage devices, show reduction of the LCOE index from 20% to 40% and improving the quality of electricity supply to the analyzed settlements.

Deterministic Asynchronous Scheduling with Probabilistic Reliability Guarantee in Industrial Wireless Networks

Industrial Wireless Networks (IWNs) play a critical role in industry 4.0 applications. However, industrial control processes require deterministic data delivery with an extremely high reliability, which is challenging in harsh IWN environments. Existing scheduling schemes rely mainly on accurate global synchronization to ensure deterministic transmission, which incurs additional hardware cost and a large communication overhead. To eliminate the synchronization requirements, we propose an asynchronous scheduling model that supports arbitrary time offset between any node pairs. As finding a solution with a given reliability guarantee in this model is a NP-hard problem, we design a probabilistic algorithm based on the Monte Carlo tree search (MCTS) method to generate and verify feasible schedules. The simulation results show that E-MCTS is significantly better in transmission reliability if compared to the classic Monte Carlo tree search algorithm.

Monte Carlo Tree Search as a Tool for Self-Learning and Teaching People to Play Complete Information Board Games

Artificial intelligence allows computer systems to make decisions similar to those of humans. However, the expert knowledge that artificial intelligence systems have is rarely used to teach non-expert humans in a specific knowledge domain. In this paper, we want to explore this possibility by proposing a tool which presents and explains recommendations for playing board games generated by a Monte Carlo Tree Search algorithm combined with Neural Networks. The aim of the aforementioned tool is to showcase the information in an easily interpretable way and to effectively transfer knowledge: in this case, which movements should be avoided, and which action is recommended. Our system displays the state of the game in the form of a tree, showing all the movements available from the current state and a set of their successors. To convince and try to teach people, the tool offers a series of queries and all information available about every possible movement. In addition, it produces a brief textual explanation for those which are recommended or not advisable. To evaluate the tool, we performed a series of user tests, observing and assessing how participants learn while using this system.

Dynamic Capacitated Arc Routing Problem in E-Bike Sharing System: A Monte Carlo Tree Search Approach

This paper studies a dynamic capacitated arc routing problem for battery replacement in an e-bike sharing system, where workers replace batteries for underpowered e-bikes along street segments dynamically. The objective is to replace as many batteries as possible and minimize pickup failures. The temporal dependency of the routing decisions, the conflict of the workers’ operations, and the stochastic and dynamic nature of user demands all make this a difficult problem. To cope with these difficulties, a “Partition-First, Route-Second” bi-level solution framework is adopted to describe the problem in two different time scales. On the large time scale, a spatiotemporal partitioning method, which divides the road network into nonoverlapping subzones, is proposed to decompose the problem. On the small time scale, this paper models the routing decision process of individual worker as a Markov Decision Process. We adopt a lookahead policy that simulates future information and decisions over some horizons to evaluate the long-term influence of current feasible decisions. A Monte Carlo Tree Search algorithm is also used to improve the simulation efficiency. By performing numerical computation experiments on a test case study and comparing with some benchmarking policies, we demonstrate the effectiveness and efficiency of the suggested method.

Development of a cyberattack response planning method for nuclear power plants by using the Markov decision process model

In an NPP cyberattack situation, not only safety response actions but also security response actions must be taken. However, operators unfamiliar with the cyber security may have difficulties in planning response actions. In this study, a security state-based cyberattack response planning method is developed using the Markov decision process model. Based on the temporal response margin analysis, available response actions are modeled as actions that can increase the available response time or decrease the response time required to secure plant safety. The response reward of an action is quantified as an increase in the response margin time. By modifying the existing action-value function and adopting the Monte-Carlo tree search algorithm, the developed method can help to establish optimal response plans that can maximize the response margin time and minimize the time required for securing plant safety. A case study validated the developed method using a hardware in the loop system.

Confidence Backup Updates for Aggregating MDP State Values in Monte-Carlo Tree Search

Monte-Carlo Tree Search (MCTS) algorithms estimate the value of MDP states based on rewards received by performing multiple random simulations. MCTS algorithms can use different strategies to aggregate these rewards and provide an estimation for the states’ values. The most common aggregation method is to store the mean reward of all simulations. Another common approach stores the best observed reward from each state. Both of these methods have complementary benefits and drawbacks. In this paper, we show that both of these methods are biased estimators for the real expected value of MDP states. We propose an hybrid approach that uses the best reward for states with low noise, and otherwise uses the mean. Experimental results on the Sailing MDP domain show that our method has a considerable advantage when the rewards are drawn from a noisy distribution.

Scalable Distributed Monte-Carlo Tree Search

Monte-Carlo Tree Search (MCTS) is remarkably successful in two-player games, but parallelizing MCTS has been notoriously difficult to scale well, especially in distributed environments. For a distributed parallel search, transposition-table driven scheduling (TDS) is known to be efficient in several domains. We present a massively parallel MCTS algorithm, that applies the TDS parallelism to the Upper Confidence bound Applied to Trees (UCT) algorithm, which is the most representative MCTS algorithm. To drastically decrease communication overhead, we introduce a reformulation of UCT called Depth-First UCT. The parallel performance of the algorithm is evaluated on clusters using up to 1,200 cores in artificial game-trees. We show that this approach scales well, achieving 740-fold speedups in the best case.

Trajectory planning for connected and automated vehicles at isolated signalized intersections under mixed traffic environment

Trajectory planning for connected and automated vehicles (CAVs) has the potential to improve operational efficiency and vehicle fuel economy in traffic systems. Despite abundant studies in this research area, most of them only consider trajectory planning in the longitudinal dimension or assume the fully CAV environment. This study proposes an approach to the decentralized planning of CAV trajectories at an isolated signalized intersection under the mixed traffic environment, which consists of connected and human-driven vehicles (CHVs) and CAVs. A bi-level optimization model is formulated based on discrete time to optimize both the longitudinal and lateral trajectories of a single CAV given signal timings and the trajectory information of surrounding vehicles. The upper-level model optimizes lateral lane-changing strategies. The lower-level model optimizes longitudinal acceleration profiles based on the lane-changing strategies from the upper-level model. Minimization of vehicle delay, fuel consumption, and lane-changing costs are considered in the objective functions. A Lane-Changing Strategy Tree (LCST) and a Parallel Monte-Carlo Tree Search (PMCTS) algorithm are designed to solve the bi-level optimization model. CAV trajectories are planned one by one according to their distance to the stop bar. A rolling horizon scheme is applied for the dynamic implementation of the proposed model with time-varying traffic conditions. Numerical studies validate the advantages of the proposed trajectory planning model compared with the benchmark cases without CAV trajectory planning. The average fuel consumption and lane-changing numbers of CAVs can be reduced noticeably, especially with high traffic demand. The delay of CAVs is reduced by ~2 s on average, which is limited due to the fixed signal timing plans. The trajectory planning of CAVs also reduces the delay and the fuel consumption of CHVs and the mixed traffic, especially with high penetration rates of CAVs. The sensitivity analysis shows that the control zone length of 200 m is sufficient to ensure the satisfactory performance of the proposed model.