Truss structure optimization via hierarchical tree search

Monte Carlo Tree Search (MCTS) is a search technique that in the last decade emerged as a major breakthrough for Artificial Intelligence applications regarding board- and video-games. In 2016, AlphaGo, an MCTS-based software agent, outperformed the human world champion of the board game Go. This game was for long considered almost infeasible for machines, due to its immense search space and the need for a long-term strategy. Since this historical success, MCTS is considered as an effective new approach for many other scientific and technical problems. Interestingly, civil structural engineering, as a discipline, offers many tasks whose solution may benefit from intelligent search and in particular from adopting MCTS as a search tool. In this work, we show how MCTS can be adapted to search for suitable solutions of a structural engineering design problem. The problem consists of choosing the load-bearing elements in a reference reinforced concrete structure, so to achieve a set of specific dynamic characteristics. In the paper, we report the results obtained by applying both a plain and a hybrid version of single-agent MCTS. The hybrid approach consists of an integration of both MCTS and classic Genetic Algorithm (GA), the latter also serving as a term of comparison for the results. The study’s outcomes may open new perspectives for the adoption of MCTS as a design tool for civil engineers.

This work explores reinforcement learning (RL) for on-board planning and scheduling of an agile Earth-observing satellite (AEOS). In this formulation of the AEOS scheduling problem, a spacecraft in low Earth orbit attempts to maximize the weighted sum of targets collected and downlinked. The AEOS scheduling problem is formulated as a Markov decision process (MDP) where the number of upcoming imaging targets included in the action space is an adjustable parameter to account for clusters of imaging targets with varying priorities. Monte Carlo tree search (MCTS) and supervised learning are used to train a set of agents with varying numbers of targets in the action space. Two backup strategies are explored for MCTS—an incremental averaging operator and a maximization operator. For all backup operators, performance asymptotically increases as the number of targets in the action space approaches the maximum number of available targets. A benchmark is computed with MCTS to determine an upper bound on performance. Furthermore, MCTS is compared with solutions generated by a genetic algorithm. MCTS demonstrates a 2%–5% increase in average reward at 10%–20% of the single-core wall clock time of the genetic algorithm. A search of various neural network hyperparameters is presented, and the trained neural networks are shown to approximate the MCTS policy with three orders of magnitude less execution time. Finally, the trained agents and the genetic algorithm are deployed on varying target densities for comparison purposes and to demonstrate robustness to mission profiles outside of the training distribution.

This study investigates the combined use of generative grammar rules and Monte Carlo tree search (MCTS) for optimizing truss structures. Our approach accommodates intermediate construction stages characteristic of progressive construction settings. We demonstrate the significant robustness and computational efficiency of our approach compared to alternative reinforcement learning frameworks from previous research activities, such as Q-learning or deep Q-learning. These advantages stem from the ability of MCTS to strategically navigate large state spaces, leveraging the upper confidence bounds for trees formula to effectively balance exploitation–exploration trade-offs. We also emphasize the importance of early decision nodes in the search tree, reflecting design choices crucial for highly performative solutions. Additionally, we show how MCTS dynamically adapts to complex and extensive state spaces without significantly affecting solution quality. While the focus of this article is on truss optimization, our findings suggest that MCTS is a powerful tool for addressing other increasingly complex engineering applications.

This research is based on the idea that certain cognitive-intensive tasks typically carried out by structural engineers—such as choosing how to effectively arrange a building’s structure—can be successfully automated. In this article we investigate two techniques widely used in the field of Artificial Intelligence, namely Monte Carlo Tree Search (MCTS) and Genetic Algorithm (GA). Following a tabula rasa approach, according to which no hints nor external data are used as a reference for navigating the search space, we demonstrate how structural designs of two-dimensional multi-storey reinforced concrete structures can be generated, with no human intervention, by implementing and combining the two techniques from a reinforcement-learning perspective. The design tasks assigned to the developed software agents concern civil structures under static and seismic loads, and the basis for comparison is represented by a combination of construction cost and greenhouse gas emissions associated with the making of the structures. In the article, based on the main concepts of Computational Mechanics, a theoretical framework is introduced, which allows to represent both structures and design tasks in a simple, analytical way. The process of gamification, to which MCTS is often associated, is then described, so that structural design is reduced to the concepts of state, actions and payoff.. Finally, case studies are presented in which different agents—based respectively on GA, MCTS, and a combination of both—are tested. The results suggest that hybrid approaches, where GA operates first followed by MCTS, are the most effective.

Monte Carlo tree search (MCTS) is a heuristic search method that is used to efficiently search decision trees. The method is particularly efficient in searching trees with a high branching factor. MCTS has a number of advantages over traditional tree search algorithms like simplicity, adaptability etc. This paper is a study of existing literature on different types of MCTS, specifically on using Genetic Algorithms with MCTS. It studies the advantages and disadvantages of this approach, and applies an enhanced variant to Gomoku, a board game with a high branching factor.

3D scene synthesis facilitates and benefits many real-world applications. Most scene generators focus on making indoor scenes plausible via learning from training data and leveraging extra constraints such as adjacency and symmetry. Although the generated 3D scenes are mostly plausible with visually realistic layouts, they can be functionally unsuitable for human users to navigate and interact with furniture. Our key observation is that human activity plays a critical role and sufficient free space is essential for human-scene interactions. This is exactly where many existing synthesized scenes fail—the seemingly correct layouts are often not fit for living. To tackle this, we present a human-aware optimization framework Haisor for 3D indoor scene arrangement via reinforcement learning, which aims to find an action sequence to optimize the indoor scene layout automatically. Based on the hierarchical scene graph representation, an optimal action sequence is predicted and performed via Deep Q-Learning with Monte Carlo Tree Search (MCTS), where MCTS is our key feature to search for the optimal solution in long-term sequences and large action space. Multiple human-aware rewards are designed as our core criteria of human-scene interaction, aiming to identify the next smart action by leveraging powerful reinforcement learning. Our framework is optimized end-to-end by giving the indoor scenes with part-level furniture layout including part mobility information. Furthermore, our methodology is extensible and allows utilizing different reward designs to achieve personalized indoor scene synthesis. Extensive experiments demonstrate that our approach optimizes the layout of 3D indoor scenes in a human-aware manner, which is more realistic and plausible than original state-of-the-art generator results, and our approach produces superior smart actions, outperforming alternative baselines.

This paper presents an innovative approach to train timetable generation using Monte Carlo tree search (MCTS) integrated with a deep reinforcement learning technique. The generation and adjustment of train timetables for high-speed railways represent a complex optimisation problem with numerous rule-based constraints that traditional mathematical methods struggle to solve efficiently. Therefore, the train timetable generation problem is modelled as a discrete spatiotemporal Markov decision process, and a comprehensive MCTS-based algorithm is developed to effectively balance exploration and exploitation through a structured tree search mechanism. The result of the comparative analysis demonstrates that MCTS-based algorithms significantly outperform state-of-the-art reinforcement learning algorithms, including double deep Q-network (DDQN) and proximal policy optimisation (PPO), achieving optimal solutions 6.5 times faster with superior training stability. To validate the scalability and real-world applicability, a large-scale case study involving 120 pairs of trains on the Beijing-Shanghai High-Speed Rail corridor over an 18-hour period successfully resolved all 45,600 initial conflicts. The optimised timetables yield significant operational improvements, including a 16.4% reduction in average delay time, 22.8% improvement in track utilisation efficiency and 9.7% reduction in energy consumption. This research contributes to the advancement of intelligent railway operations optimisation and demonstrates the potential of MCTS-based approaches to transform complex transportation problems.

This paper introduces a novel enhancement for Monte Carlo Tree Search (MCTS) which consists of using a Genetic Algorithm (GA) to evolve macro-actions for a particular domain. A macro-action is a sequence of actions treated as a single action; using this approach MCTS searches over a set of macro-actions rather than the domain's default actions. If macro-actions are appropriate for a domain their use increases search tree depth and avoids unproductive actions. This is particularly useful in video game domains where it is often prohibitively expensive to build a sufficiently deep search tree for effective play due to expensive simulations, a small amount of computation time per action, and action spaces where each individual action usually has a very small effect on the state. Macro-action approaches have previously been shown to address look-ahead issues and improve MCTS performance in video game domains. However, previous implementations have relied on hand-designing very simple macro-actions. Our evolutionary approach automates this design process and significantly improves MCTS performance in the game Flappy Bird and several Atari 2600 games.

This work explores on-board planning for the single spacecraft, multiple ground station Earth-observing satellite scheduling problem through artificial neural network function approximation of state–action value estimates generated by Monte Carlo tree search (MCTS). An extensive hyperparameter search is conducted for MCTS on the basis of performance, safety, and downlink opportunity utilization to determine the best hyperparameter combination for data generation. A hyperparameter search is also conducted on neural network architectures. The learned behavior of each network is explored, and each network architecture’s robustness to orbits and epochs outside of the training distributions is investigated. Furthermore, each algorithm is compared with a genetic algorithm, which serves to provide a baseline for optimality. MCTS is shown to compute near-optimal solutions in comparison to the genetic algorithm. The state–action value networks are shown to match or exceed the performance of MCTS in six orders of magnitude less execution time, showing promise for execution on board spacecraft.

Reinforcement learning (RL) algorithms that include Monte Carlo Tree Search (MCTS) have found tremendous success in computer games such as Go, Shiga and Chess. Such learning algorithms have demonstrated super-human capabilities in navigating through an exhaustive discrete action search space. Motivated by their success in computer games, we demonstrate that RL can be applied to inverse materials design problems. We deploy RL for a representative case of the optimal atomic scale inverse design of extended defects via rearrangement of chalcogen (e.g. S) vacancies in 2D transition metal dichalcogenides (e.g. MoS2). These defect rearrangements and their dynamics are important from the perspective of tunable phase transition in 2D materials i.e. 2H (semi-conducting) to 1T (metallic) in MoS2. We demonstrate the ability of MCTS interfaced with a reactive molecular dynamics simulator to efficiently sample the defect phase space and perform inverse design—starting from randomly distributed S vacancies, the optimal defect rearrangement of defects corresponds a line defect of S vacancies. We compare MCTS performance with evolutionary optimization i.e. genetic algorithms and show that MCTS converges to a better optimal solution (lower objective) and in fewer evaluations compared to GA. We also comprehensively evaluate and discuss the effect of MCTS hyperparameters on the convergence to solution. Overall, our study demonstrates the effectives of using RL approaches that operate in discrete action space for inverse defect design problems.

Teams of agents offer the potential to defeat higher-performance adversaries by employing intelligent coordination. This work examines how Monte Carlo tree search (MCTS) can be utilized for multi-agent, six-degree-of-freedom (6-DoF) engagements. This work introduces a novel approach to action space selection for known MCTS algorithm frameworks. This enables intelligent coordination for 6-DoF multi-agent adversarial game environments with continuous state and action spaces. Our framework leverages a genetic algorithm to determine subsets of algorithmically generated actions to overcome disadvantaged situations. We quantify the benefits of our proposed approach by examining multi-agent aerial engagements with F-16s and P-51Ds. We compare our approach to uncoordinated pure pursuit and reinforcement learning agents. Our results illustrate that the proposed genome-action-informed MCTS substantially increases the win rate compared to pure pursuit and baseline MCTS. These results hold across both aircraft platforms. In addition, we show how our algorithm can achieve high win rates even when operating at a large maneuverability or speed disadvantage. These results illustrate the benefits of MCTS for coordinating 6-DoF engagements and highlight the benefits of intelligent action space selection.

Gobang is a worldwide two-player strategy board game, which is popular especially in the Asian region. Recently, the science and technology has been developing fast and artificial intelligence was applied in numerous fields like the board game. Gobang, a strategy game with moderate difficulty, is a suitable example for people to test algorithms and solve board game problems. As a result, many related algorithms were appearing with different advantages and disadvantages. In this work, these already existed algorithms, game tree, minimax search, alpha-beta pruning, genetic algorithm and monte carlo tree search, were discussed and compared. The results and comparison showed that game tree and minimax search had a large number of nodes to calculate to reach a suitable search depth, about 1.00E+12 and 1.29E+14 respectively, which meant they need a long calculating time, while the alpha-beta pruning need to calculate about 2.2E+07 nodes and genetic algorithm only need to calculate about 1.00E+04 nodes, which cost 0.6 seconds for every move. Plus, the monte carlo tree search could reach nearly 100% win rate through self-play, which making gobang algorithm become more refined. Additionally, these algorithms had already made gobang AI powerful with fast move and high win rate, so they also had been applied in many different fields to develop and spread gobang.

In the realm of medicinal chemistry, the predominant challenge in molecular design lies in managing extensive molecular data sets and effectively screening for, as well as preserving, molecules with potential value. Traditional methodologies typically utilize deep learning models or genetic algorithms (GA) for optimization, yet each approach has inherent limitations: deep learning models are constrained by substantial computational resource demands; genetic algorithms often yield molecular structures with low validity and feasibility. To overcome these challenges, we have developed the Molecular multiobjective optimization of Monte Carlo Tree Search (MCTS) and Non-Superiority Ranking Genetic Algorithm II (NSGA-II)-MNopt, which ingeniously integrates MCTS with NSGA-II. Specifically, NSGA-II demonstrates unique strengths in balancing multiple optimization objectives and achieves rapid performance through its crowding distance and nondominated ordering mechanisms, while MCTS focuses on enhancing the validity of molecular structures to ensure that the generated molecules are both desirable and feasible. Notably, MNopt does not require reliance on extensive molecular training data sets in the initial stages, effectively mitigating excessive resource consumption. Experimental results demonstrate that MNopt surpasses existing techniques in multiobjective optimization, generating effective and diverse molecular structures, thereby offering a crucial tool for novel drug discovery and materials science.

The dynamic flexible job shop scheduling problem (DFJSP) with machine faults, considering the recovery condition and variable processing time, is studied to determine the rescheduling scheme when machine faults occur in real time. The Monte Carlo Tree Search (MCTS) algorithm with reinforcement learning and the relational-enhanced graph attention network (MGRL) is presented to address the DFJSP with machine faults, considering the recovery condition and variable processing time. The MCTS with the skip-node restart strategy, which utilizes local optimal solutions found during the Monte Carlo sampling process, is designed to enhance the optimization efficiency of MCTS in real time. A relational graph attention network (RGAT), a relational-enhanced and transformer-integrated graph network in the MGRL, is designed to analyze the scheduling disjunctive graph, guide the Monte Carlo sampling method to improve sampling efficiency, and enhance the quality of MCTS optimization decisions. Experimental results demonstrate the effectiveness of the RGAT and the skip-node restart strategy. Further application analysis results show that the MGRL is optimal among all comparison methods when algorithms solve the DFJSP.

In this article we explore the advantages, and disadvantages, of solving multi-agent to task assignment problems by searching state-space trees using Monte Carlo Tree Search (MCTS) and Process Algebra (PA). State-Space trees have been used for many years to assign tasks to agents. Since these trees exhibit exponential growth as the problem size increases, optimal solutions can be hard to find. Over the years we have used Depth First Branch and Bound (DFBnB), and Genetic Algorithms (GA) to find solutions, but each of these methods have drawbacks. Introduction of MCTS has the potential to alleviate these drawbacks while improving solution quality. Furtermore, the integration of Process Algebra with MCTS allows for richer problem formulations by making it possible to define relationships between generic tasks.