Monte Carlo Tree Search Method Research Articles

Optimizing Camera Motion with MCTS and Target Motion Modeling in Multi-Target Active Object Tracking

In this work, we are dedicated to multi-target active object tracking (AOT), where the goal is to achieve continuous tracking of targets through real-time control of camera. This form of active camera control can be applied to unmanned aerial vehicles (UAV), intelligent robots, and sports events. Our work is conducted in an environment featuring multiple cameras and targets, where our goal is to maximize target coverage. Contrasting with previous research, our work introduces additional degrees of freedom for the cameras, allowing them not only to rotate but also to move along boundary lines. In addition, we model the motion of target to predict the future position of the target in environment. With target’s future position, we use Monte Carlo Tree Search (MCTS) method to find the optimal action of camera. Since the action space is large, we propose to leverage the action selection from multi-agent reinforcement learning (MARL) network to prune the search tree of Monte Carlo Tree Search method, so as to find the optimal action more efficiently. We establish a multi-target 2D environment to simulate several sports games, and experimental results demonstrate that our method can effectively improve the target coverage. The code is available at: http://github.com/HopeChanger/ActiveObjectTracking .

3D/2D Vessel Registration Based on Monte Carlo Tree Search and Manifold Regularization.

The augmented intra-operative real-time imaging in vascular interventional surgery, which is generally performed by projecting preoperative computed tomography angiography images onto intraoperative digital subtraction angiography (DSA) images, can compensate for the deficiencies of DSA-based navigation, such as lack of depth information and excessive use of toxic contrast agents. 3D/2D vessel registration is the critical step in image augmentation. A 3D/2D registration method based on vessel graph matching is proposed in this study. For rigid registration, the matching of vessel graphs can be decomposed into continuous states, thus 3D/2D vascular registration is formulated as a search tree problem. The Monte Carlo tree search method is applied to find the optimal vessel matching associated with the highest rigid registration score. For nonrigid registration, we propose a novel vessel deformation model based on manifold regularization. This model incorporates the smoothness constraint of vessel topology into the objective function. Furthermore, we derive simplified gradient formulas that enable fast registration. The proposed technique undergoes evaluation against seven rigid and three nonrigid methods using a variety of data - simulated, algorithmically generated, and manually annotated - across three vascular anatomies: the hepatic artery, coronary artery, and aorta. Our findings show the proposed method's resistance to pose variations, noise, and deformations, outperforming existing methods in terms of registration accuracy and computational efficiency. The proposed method demonstrates average registration errors of 2.14 mm and 0.34 mm for rigid and nonrigid registration, and an average computation time of 0.51 s.

Beyond games: a systematic review of neural Monte Carlo tree search applications

The advent of AlphaGo and its successors marked the beginning of a new paradigm in playing games using artificial intelligence. This was achieved by combining Monte Carlo tree search, a planning procedure, and deep learning. While the impact on the domain of games has been undeniable, it is less clear how useful similar approaches are in applications beyond games and how they need to be adapted from the original methodology. We perform a systematic literature review of peer-reviewed articles detailing the application of neural Monte Carlo tree search methods in domains other than games. Our goal is to systematically assess how such methods are structured in practice and if their success can be extended to other domains. We find applications in a variety of domains, many distinct ways of guiding the tree search using learned policy and value functions, and various training methods. Our review maps the current landscape of algorithms in the family of neural monte carlo tree search as they are applied to practical problems, which is a first step towards a more principled way of designing such algorithms for specific problems and their requirements.

A 3D Monte Carlo tree search method for railway alignment optimization

Railway alignment design is an important process, which fundamentally affects the construction, operation and maintenance of a railway. However, optimizing an alignment is challenging due to, e.g., the usually huge search space, infinite number of possible alternatives and numerous constraints. To address this problem, we propose a three-dimensional Monte Carlo Tree Search (3D-MCTS) method for alignment optimization. Specifically, a time-varying selection approach is first designed for efficiently exploring the search space. Then, the feasible search space is dynamically delineated with a customized tree expansion operator to accelerate the search process. In addition, a simulation strategy with global reward estimation is proposed to balance global exploration and local exploitation during optimization, which contributes to enhancing the quality of the optimized alignment. Finally, the proposed 3D-MCTS is applied to a complex real-world railway case. It shows that the 3D-MCTS can find better solutions compared to the best alignments that are manually designed by experienced engineers or produced by a previous distance transform algorithm. Two sensitivity analyses also reveal the 3D-MCTS's performance and robustness with respect to cost optimization and search efficiency.

Multi-UAV Urban Logistics Task Allocation Method Based on MCTS

Unmanned aerial vehicles (UAVs) open new methods for efficient and rapid transportation in urban logistics distribution, where task allocation is a significant issue. In urban logistics systems, the energy status of UAVs is a critical factor in ensuring mission fulfillment. While extensive literature addresses the energy consumption of UAVs during tasks, the feasibility of energy replenishment must be addressed, which introduces additional uncertainty to the task allocation. This paper realizes multi-tasking, considering the energy consumption and replenishment of UAVs, to ensure that the tasks can be accomplished while reducing energy consumption. This paper proposes uniform distribution K-means to realize balanced multi-task grouping. Based on the Monte Carlo tree search (MCTS), a task-allocation-oriented MCTS method is proposed, including improving the selection and simulation process of MCTS. The aim was to collaborate with multiple trees for node selection and record historical simulation information to guide subsequent simulations for better results. Finally, the optimality of the proposed method was validated by comparing it with other relevant MCTS methods through several randomized experiments.

A policy-based Monte Carlo tree search method for container pre-marshalling

The container pre-marshalling problem (CPMP) aims to minimise the number of reshuffling moves, ultimately achieving an optimised stacking arrangement in each bay based on the priority of containers during the non-loading phase. Given the sequential decision nature, we formulated the CPMP as a Markov decision process (MDP) model to account for the specific state and action of the reshuffling process. To address the challenge that the relocated container may trigger a chain effect on the subsequent reshuffling moves, this paper develops an improved policy-based Monte Carlo tree search (P-MCTS) to solve the CPMP, where eight composite reshuffling rules and modified upper confidence bounds are employed in the selection phases, and a well-designed heuristic algorithm is utilised in the simulation phases. Meanwhile, considering the effectiveness of reinforcement learning methods for solving the MDP model, an improved Q-learning is proposed as the compared method. Numerical results show that the P-MCTS outperforms all compared methods in scenarios where all containers have different priorities and scenarios where containers can share the same priority.

Codebook Based Antenna Configuration: A New Network Planning Paradigm for mmWave Mobile Communication Systems

To configure the azimuths and downtilts of massive antennas optimally is of great importance for the mmWave mobile communication systems to support diverse services, however, it is always computationally forbidden due to the huge number of possible combinations in real-world mobile networks. In this paper, we develop an effective and efficient method to deal with this intractable optimization task. First, we explore the most promising azimuth and downtilt configurations of each antenna with a branch-and-bound procedure followed by a semi-definite relaxation algorithm, which finally generates a codebook with finite elements to reduce the unnecessary searches for similar or unpromising configurations during global search; second, a Monte Carlo tree search method is introduced to iteratively optimize the azimuths and downtilts of all antennas, aiming at alleviating the mutual interference of adjacent antennas to maximize the signal-to-interference ratio of the target service area, during which the generated antenna configuration codebook guides the iterative search process efficiently. Experiment results in real urban scenarios show that our proposed method can produce the best signal-to-interference ratio coverage as compared with state-of-the-art ones. Moreover, the proposed algorithm can scale up straightforwardly, making it a competitive choice for large-scale network optimization.

Adversarial agent-learning for cybersecurity: a comparison of algorithms

Abstract We investigate artificial intelligence and machine learning methods for optimizing the adversarial behavior of agents in cybersecurity simulations. Our cybersecurity simulations integrate the modeling of agents launching Advanced Persistent Threats (APTs) with the modeling of agents using detection and mitigation mechanisms against APTs. This simulates the phenomenon of how attacks and defenses coevolve. The simulations and machine learning are used to search for optimal agent behaviors. The central question is: under what circumstances, is one training method more advantageous than another? We adapt and compare a variety of deep reinforcement learning (DRL), evolutionary strategies (ES) and Monte Carlo Tree Search methods within Connect 4, a baseline game environment, and on both a simulation supporting a simple APT threat model, SNAPT, as well as CyberBattleSim, an open-source cybersecurity simulation. Our results show that when attackers are trained by DRL and ES algorithms, as well as when they are trained with both algorithms being used in alternation, they are able to effectively choose complex exploits that thwart a defense. The algorithm that combines DRL and ES achieves the best comparative performance when attackers and defenders are simultaneously trained, rather than when each is trained against its non-learning counterpart.

Robot navigation with predictive capabilities using graph learning and Monte Carlo tree search

This article develops a prediction and path planning system based on the graph neural network to navigate a robot in a complex dynamic environment. In particular, the core of this method is to predict those aspects of the future that are directly relevant for planning, including their value, state, and policy. A graph neural network-based method is introduced to encode the interaction between the robot and the surrounding environment. Then, the dynamic model of the environment is learned through the model-based reinforcement learning, and the path is planned using the Monte Carlo tree search method according to the learned model. Finally, simulation studies are given to evaluate the validity and advantage of the obtained algorithm compared with the most recent methods. It has been shown that the proposed method achieves a higher success rate within a less time. Meantime, the oscillatory and freezing problems caused by the short-sightedness of the robot are avoided.

A Monte Carlo tree search conceptual framework for feature model analyses

Challenging domains of the future such as Smart Cities, Cloud Computing, or Industry 4.0 expose highly variable systems with colossal configuration spaces. The automated analysis of those systems’ variability has often relied on SAT solving and constraint programming. However, many of the analyses have to deal with the uncertainty introduced by the fact that undertaking an exhaustive exploration of the whole configuration space is usually intractable. In addition, not all analyses need to deal with the configuration space of the feature models, but with different search spaces where analyses are performed over the structure of the feature models, the constraints, or the implementation artifacts, instead of configurations. This paper proposes a conceptual framework that tackles various of those analyses using Monte Carlo tree search methods, which have proven to succeed in vast search spaces (e.g., game theory, scheduling tasks, security, program synthesis, etc.). Our general framework is formally described, and its flexibility to cope with a diversity of analysis problems is discussed. We provide a Python implementation of the framework that shows the feasibility of our proposal, identifying up to 11 lessons learned, and open challenges about the usage of the Monte Carlo methods in the software product line context. With this contribution, we envision that different problems can be addressed using Monte Carlo simulations and that our framework can be used to advance the state-of-the-art one step forward.

Scalable Online Planning for Multi-Agent MDPs

We present a scalable tree search planning algorithm for large multi-agent sequential decision problems that require dynamic collaboration. Teams of agents need to coordinate decisions in many domains, but naive approaches fail due to the exponential growth of the joint action space with the number of agents. We circumvent this complexity through an approach that allows us to trade computation for approximation quality and dynamically coordinate actions. Our algorithm comprises three elements: online planning with Monte Carlo Tree Search (MCTS), factored representations of local agent interactions with coordination graphs, and the iterative Max-Plus method for joint action selection. We evaluate our approach on the benchmark SysAdmin domain with static coordination graphs and achieve comparable performance with much lower computation cost than our MCTS baselines. We also introduce a multi-drone delivery domain with dynamic coordination graphs, and demonstrate how our approach scales to large problems on this domain that are intractable for other MCTS methods. We provide an open-source implementation of our algorithm at https://github.com/JuliaPOMDP/FactoredValueMCTS.jl.

Method for assessing the characteristics of the quality of service of networks

Variational quantum algorithms are at the forefront of modeling fault-tolerant quantum devices of the near future. While most variational quantum algorithms use only continuous optimization variables, their representative power can sometimes be greatly increased by adding certain discrete optimization variables, as illustrated by the example of the generalized quantum approximation optimization algorithm. However, the hybrid discrete-continuous optimization problem poses an optimization problem. The paper proposes a new algorithm that combines the Monte Carlo tree search method with an improved gradient solver to optimize discrete and continuous variables in a quantum circuit, respectively. The algorithm has been found to have excellent noise tolerance properties and is superior to prior algorithms in the field.

Self-play learning strategies for resource assignment in Open-RAN networks

Open Radio Access Network (ORAN) is being developed with an aim to democratise access and lower the cost of future mobile data networks, supporting network services with various QoS requirements, such as massive IoT and URLLC. In ORAN, network functionality is dis-aggregated into remote units (RUs), distributed units (DUs) and central units (CUs), which allows flexible software on Commercial-Off-The-Shelf (COTS) deployments. Furthermore, the mapping of variable RU requirements to local mobile edge computing centres for future centralised processing would significantly reduce the power consumption in cellular networks. In this paper, we study the RU–DU resource assignment problem in an ORAN system, modelled as a 2D bin packing problem. A deep reinforcement learning-based self-play approach is proposed to achieve efficient RU–DU resource management, with AlphaGo Zero inspired neural Monte-Carlo Tree Search (MCTS). Experiments on representative 2D bin packing environment and real sites data show that the self-play learning strategy achieves intelligent RU–DU resource assignment for different network conditions. Comparing with baseline methods, including a heuristic virtual resource allocation algorithm, the Lego heuristic algorithm and the MCTS methods, the proposed approach achieves a performance gain between 5.70% to 12.95% in terms of resource utilisation efficiency.

A Defense Strategy Selection Method Based on the Cyberspace Wargame Model

Network defenders always face the problem of how to use limited resources to make the most reasonable decision. The network attack-defense game model is an effective means to solve this problem. However, existing network attack-defense game models usually assume that defenders will no longer change defense strategies after deploying them. However, in an advanced network attack-defense confrontation, defenders usually redeploy defense strategies for different attack situations. Therefore, the existing network attack-defense game models are challenging to accurately describe the advanced network attack-defense process. To address the above challenges, this paper proposes a defense strategy selection method based on the network attack-defense wargame model. We model the advanced network attack-defense confrontation process as a turn-based wargame in which both attackers and defenders can continuously adjust their strategies in response to the attack-defense posture and use the Monte Carlo tree search method to solve the optimal defense strategy. Finally, a network example is used to illustrate the effectiveness of the model and method in selecting the optimal defense strategy.

UAV Intelligent Coverage Navigation Based on DRL in Complex Geometrical Environments

Unmanned aerial vehicle (UAV) is one of the preferred tools for coverage detection missions, because of its maneuverability and flexibility. It is challenging for the UAV to decide a track by itself in a complex geometrical environment. This paper presents a UAV intelligent navigation method based on deep reinforcement learning (DRL). We propose using geographic information systems (GIS) as the DRL training environment to overcome the inconsistency between the training environment and the test environment. We creatively save the flight path in the form of an image. The combination of the knowledge-based Monte Carlo tree search method and local search method can not only effectively avoid falling into local search, but also ensure learning the optimal search direction under the limitation of computing power. Experiments show that the trained UAV can find an excellent flight path by intelligent navigation, and able to make effective flight decisions in a complex geometrical environment.

Stochastic Motion Planning Under Partial Observability for Mobile Robots With Continuous Range Measurements

In this article, we address the problem of stochastic motion planning under partial observability, more specifically, how to navigate a mobile robot equipped with continuous range sensors, such as LIDAR. In contrast to many existing robotic motion planning methods, we explicitly consider the uncertainty of the robot state by modeling the system as a partially observable Markov decision process (POMDP). Recent work on general purpose POMDP solvers is typically limited to discrete observation spaces, and does not readily apply to the proposed problem due to the continuous measurements from LIDAR. In this article, we build upon an existing Monte Carlo tree search method, partially observable Monte Carlo planning (POMCP), and propose a new algorithm POMCP++. Our algorithm can handle continuous observation spaces with a novel measurement selection strategy. The POMCP++ algorithm overcomes overoptimism in the value estimation of a rollout policy by removing the implicit perfect state assumption at the rollout phase. We validate POMCP++ in theory by proving it is a Monte Carlo tree search algorithm. Through comparisons with other methods that can also be applied to the proposed problem, we show that POMCP++ yields significantly higher success rate and total reward.

Neural networks and Monte-Carlo method usage in multi-agent systems for sudoku problem solving

The object of research is multi-agent systems based on Deep Reinforcement Learning algorithms and analysis of ways to establish interaction within the system, based on intelligent agents. Also, part of the material in this paper covers ways to organize the management and administration of agents at the meta-level: external controllers and tools to optimize their work, describing architectural solutions that should accelerate agents’ training. The studied full-fledged multi-agent system would be flexible to expansion and would give effective acceleration in agent training and problem-solving quality.In this paper, the following neural network models were considered: DQN, DDQN, PPO, TD (methods based on Q-Learning), an approach using a neural network with Monte-Carlo tree search. The presented models were tested on a Sudoku problem with a dataset of 5039 combinations, dimensions 2x2, 4x4, and 9x9. Several sets of agent rewards were used. The presentation of data during the learning and problem-solving process was described. Also was built a multi-agent system based on the model using a Monte-Carlo tree search.According to the study results, it was revealed that for tasks in a complex environment, the models based on Q-Learning are practically ineffective (plots support the statement). The training process for these models is quite demanding on the characteristics of the workstation hardware. It was also determined that the Monte-Carlo tree search method does a good job. Even with a small number of iterations, it shows results better than other Deep Learning methods (45–50 % accuracy for 9x9). However, a significant drawback is a complexity of training the model, and the hardware requirements are too large for this kind of research.

The Concept of Constructing an Artificial Dispatcher Intelligent System Based on Deep Reinforcement Learning for the Automatic Control System of Electric Networks

The concept of the Artificial Dispatcher autonomous intelligent control system for integrating into modern automatic control systems of electric networks in order to improve the efficiency of managing the modes of electric power systems is proposed. This intelligent control system is implemented based on the deep machine learning technology and joint use of the Monte Carlo tree search method and deep artificial neural networks. It is shown that the efficient training of the Artificial Dispatcher intelligent control system is achievable through an innovative self-play algorithm, without using large databases of circuit-mode situations and expertise in managing the modes of electric power systems. Calculation examples of using agents created based on the concept of the Artificial Dispatcher intelligent control system in the problems of the automation of substation technological processes (managing voltage and reactive power controllers) and industrial enterprises (DC servomotor control) are presented.

Design of a Reinforcement Learning-Based Lane Keeping Planning Agent for Automated Vehicles

Reinforcement learning-based approaches are widely studied in the literature for solving different control tasks for Connected and Autonomous Vehicles, from which this paper deals with the problem of lateral control of a dynamic nonlinear vehicle model, performing the task of lane-keeping. In this area, the appropriate formulation of the goals and environment information is crucial, for which the research outlines the importance of lookahead information, enabling to accomplish maneuvers with complex trajectories. Another critical part is the real-time manner of the problem. On the one hand, optimization or search based methods, such as the presented Monte Carlo Tree Search method, can solve the problem with the trade-off of high numerical complexity. On the other hand, single Reinforcement Learning agents struggle to learn these tasks with high performance, though they have the advantage that after the training process, they can operate in a real-time manner. Two planning agent structures are proposed in the paper to resolve this duality, where the machine learning agents aid the tree search algorithm. As a result, the combined solution provides high performance and low computational needs.

A single-player Monte Carlo tree search method combined with node importance for virtual network embedding

As a critical technology in network virtualization, virtual network embedding (VNE) focuses on how to allocate physical resources to virtual network requests efficiently. Because the VNE problem is NP-hard, most of the existing approaches are heuristic-based algorithms that tend to converge to a local optimal solution and have a low performance. In this paper, we propose an algorithm that combines the basic Monte Carlo tree search (MCTS) method with node importance to apply domain-specific knowledge. For a virtual network request, we first model the embedding process as a finite Markov decision process (MDP), where each virtual node is embedded in one state in the order of node importance that we design. The shortest-path algorithm is then applied to embed links in the terminal state and return the cost as a part of the reward. Due to the reward delay mechanism of the MDP, the result of link mapping can affect the action selected in the previous node mapping stage, coordinating the two embedding stages. With node importance, domain-specific knowledge can be used in the Expansion and Simulation stages of MCTS to speed up the search and estimate the simulation value more accurately. The experimental results show that, compared with the existing classic algorithms, our proposed algorithm can improve the performance of VNE in terms of the average physical node utilization ratio, acceptance ratio, and long-term revenue to cost ratio.