Lookahead Search Research Articles

Efficient retrosynthetic planning with MCTS exploration enhanced A* search

Retrosynthetic planning, which aims to identify synthetic pathways for target molecules from starting materials, is a fundamental problem in synthetic chemistry. Computer-aided retrosynthesis has made significant progress, in which heuristic search algorithms, including Monte Carlo Tree Search (MCTS) and A* search, have played a crucial role. However, unreliable guiding heuristics often cause search failure due to insufficient exploration. Conversely, excessive exploration also prevents the search from reaching the optimal solution. In this paper, MCTS exploration enhanced A* (MEEA*) search is proposed to incorporate the exploratory behavior of MCTS into A* by providing a look-ahead search. Path consistency is adopted as a regularization to improve the generalization performance of heuristics. Extensive experimental results on 10 molecule datasets demonstrate the effectiveness of MEEA*. Especially, on the widely used United States Patent and Trademark Office (USPTO) benchmark, MEEA* achieves a 100.0% success rate. Moreover, for natural products, MEEA* successfully identifies bio-retrosynthetic pathways for 97.68% test compounds.

Deep Reinforcement Learning With a Look-Ahead Search for Robotic Cell Scheduling

Deep Reinforcement Learning With a Look-Ahead Search for Robotic Cell Scheduling

Searching for answers: expert pattern recognition and planning.

Does expertise mostly stem from pattern recognition or look-ahead search? van Opheusden et al. contribute to this important debate in cognitive psychology and artificial intelligence (AI) with a multi-method, multi-experiment study and a new model. Using a novel, relatively simple board game, they show that players increase depth of search when improving their skill.

Look-ahead based reinforcement learning for robotic flow shop scheduling

Scheduling of a robotic flow shop, where a dual-gripper robot transports parts between machines, is addressed with a makespan measure. Most previous studies on robotic flow shop scheduling have focused on cyclic scheduling where the robot repeats a certain sequence to process identical parts. Recently, noncyclic scheduling of robotic flow shops is strongly required due to the need for producing customized products and smaller order sizes efficiently. This study therefore considers noncyclic scheduling of a dual-gripper robotic flow shop with a given part sequence and proposes a novel solution approach, look-ahead based reinforcement learning (LARL). The LARL consists of deep Q-learning for training a Q-network based on a given set of instances and the look-ahead search used for testing new instances. The look-ahead search in the LARL is efficient, especially for robotic flow shop scheduling where future state information can be used for determining the current robot task. The experimental results comparing the LARL with an optimal algorithm and the well-known robot task sequence show the effectiveness of the LARL.

Lookahead Search-Based Low-Complexity Multi-Type Tree Pruning Method for Versatile Video Coding (VVC) Intra Coding

The latest versatile video coding (VVC) introduces some novel techniques such as quadtree with nested multi-type tree (QTMT), multiple transform selection (MTS) and multiple reference line (MRL). These tools improve compression efficiency compared with the previous standard H.265/HEVC, but they suffer from very high computational complexity. One of the most time-consuming parts of VVC intra coding is the coding tree unit (CTU) structure decision. In this paper, we propose a low-complexity multi-type tree (MT) pruning method for VVC intra coding. This method consists of lookahead search and MT pruning. The lookahead search process is performed to derive the approximate rate-distortion (RD) cost of each MT node at depth 2 or 3. Subsequently, the improbable MT nodes are pruned by different strategies under different cost errors. These strategies are designed according to the priority of the node. Experimental results show that the overall proposed algorithm can achieve 47.15% time saving with only 0.93% Bjøntegaard delta bit rate (BDBR) increase over natural scene sequences, and 45.39% time saving with 1.55% BDBR increase over screen content sequences, compared with the VVC reference software VTM 10.0. Such results demonstrate that our method achieves a good trade-off between computational complexity and compression quality compared to recent methods.

Enhancing Feedback Steering Controllers for Autonomous Vehicles With Deep Monte Carlo Tree Search

Steering control is a vital function for autonomous vehicles, whose performance largely determines the driving safety. The widely used feedback steering controllers degrade significantly when vehicles drive at high speeds. This is mainly because these controllers can not effectively exploit nonlinear vehicle dynamics and future road information. Hence, the steering actions generated by these controllers can not achieve accurate and stable path tracking performance. To overcome this limitation, we propose a deep learning based Monte Carlo Tree Search (DeepMCTS) method to enhance the feedback controllers. Due to the accuracy of the learning-based model in characterizing nonlinear vehicle dynamics and the lookahead search ability of MCTS, the control performance of the feedback controllers can be enhanced significantly. Different from MPC methods, which leverage vehicle dynamics and future road information via online optimization, our method relies on the efficient tree search process. Hence, our method can avoid heavy computational burden. Testing results validate the proposed method can achieve better tracking accuracy and stability in high speeds scenarios.

AlphaDrug: protein target specific de novo molecular generation.

Traditional drug discovery is very laborious, expensive, and time-consuming, due to the huge combinatorial complexity of the discrete molecular search space. Researchers have turned to machine learning methods for help to tackle this difficult problem. However, most existing methods are either virtual screening on the available database of compounds by protein-ligand affinity prediction, or unconditional molecular generation, which does not take into account the information of the protein target. In this paper, we propose a protein target-oriented de novo drug design method, called AlphaDrug. Our method is able to automatically generate molecular drug candidates in an autoregressive way, and the drug candidates can dock into the given target protein well. To fulfill this goal, we devise a modified transformer network for the joint embedding of protein target and the molecule, and a Monte Carlo tree search (MCTS) algorithm for the conditional molecular generation. In the transformer variant, we impose a hierarchy of skip connections from protein encoder to molecule decoder for efficient feature transfer. The transformer variant computes the probabilities of next atoms based on the protein target and the molecule intermediate. We use the probabilities to guide the look-ahead search by MCTS to enhance or correct the next-atom selection. Moreover, MCTS is also guided by a value function implemented by a docking program, such that the paths with many low docking values are seldom chosen. Experiments on diverse protein targets demonstrate the effectiveness of our methods, indicating that AlphaDrug is a potentially promising solution to target-specific de novo drug design.

Post-storm repair crew dispatch for distribution grid restoration using stochastic Monte Carlo tree search and deep neural networks

Natural disasters such as storms usually bring significant damages to distribution grids. This paper investigates the optimal routing of utility vehicles to restore outages in the distribution grid as fast as possible after a storm. First, the post-storm repair crew dispatch task with multiple utility vehicles is formulated as a sequential stochastic optimization problem. In the formulated optimization model, the belief state of the power grid is updated according to the phone calls from customers and the information collected by utility vehicles. Second, an AlphaZero based utility vehicle routing (AlphaZero-UVR) approach is developed to achieve the real-time dispatching of the repair crews. The proposed AlphaZero-UVR approach combines stochastic Monte-Carlo tree search (MCTS) with deep neural networks to give a lookahead search decisions, which can learn to navigate repair crews without human guidance. Simulation results show that the proposed approach can efficiently navigate crews to repair all outages.

When to Commit to an Action in Online Planning and Search

In online planning, search is concurrent with execution. Under the formulation of planning as heuristic search, when a planner commits to an action, it re-roots its search tree at the node representing the outcome of that action. For the system to remain controlled, the planner must commit to a new action (perhaps a no-op) before the previously chosen action completes. This time pressure results in a real-time search. In this time-bounded setting, it can be beneficial to commit early, in order to perform more lookahead search focused below an upcoming state. In this paper, we propose a principled method for making this commitment decision. Our experimental evaluation shows that our scheme can outperform previously-proposed fixed strategies.

Design and Testing of Headland Turning Algorithms Based on Transition Distance Prediction for Autonomous Rice Transplanter

Highlights Autonomous rice transplanters often deviate during the transition between turning and straight paths. Headland turning path planning algorithms were designed to reduce transition distances before and after turning. Transition distances were predicted based on lateral deviations and heading errors over a threshold using simulations. The developed algorithms were validated for reducing transition distance with an auto-guidance system in a paddy field. Abstract. In the straight sections before and after headland turning in paddy fields, high lateral deviations and heading errors may occur due to changes in the path curvature and dynamic factors of the vehicle and path tracking algorithm. In the case of autonomous rice transplanters, such errors may lengthen the transitions between turning and straight sections and deteriorate working efficiency. In this study, headland turning path generation algorithms for rice transplanters were proposed to reduce the transition distances. Path models were defined considering the transition distances to move the vehicle such that the error was within the acceptable range. The transition distances were predicted by using a simulator that employed a kinematic vehicle model and a tracking algorithm with a model-based control system for generating steering angles. A look-ahead search method and tilt compensation for the Global Navigation Satellite System were applied as well. The developed method was validated in a paddy field by using a Daedong DRP60D rice transplanter. The transition distance decreased by 34.04% compared to that associated with the baseline path that did not consider the transitions. The overlap area decreased by an average of 55.51% within the straight lines. During straight operation, the maximum lateral deviation did not exceed 15 cm. These results confirmed that the proposed algorithms for autonomous rice transplanters can traverse a target path more accurately before and after turning and, consequently, increase work efficiency by avoiding non-planted areas. Keywords: Auto-guidance, Headland turning, Path planning, Prediction, Rice transplanter, Transition.

Puppet Search: Enhancing Scripted Behavior by Look-Ahead Search with Applications to Real-Time Strategy Games

Real-Time Strategy (RTS) games have shown to be very resilient to standard adversarial tree search techniques. Recently, a few approaches to tackle their complexity have emerged that use game state or move abstractions, or both. Unfortunately, the supporting experiments were either limited to simpler RTS environments (uRTS, SparCraft) or lack testing against state-of-the-art game playing agents. Here, we propose Puppet Search, a new adversarial search framework based on scripts that can expose choice points to a look-ahead search procedure. Selecting a combination of a script and decisions for its choice points represents a move to be applied next. Such moves can be executed in the actual game, thus letting the script play, or in an abstract representation of the game state which can be used by an adversarial tree search algorithm. Puppet Search returns a principal variation of scripts and choices to be executed by the agent for a given time span. We implemented the algorithm in a complete StarCraft bot. Experiments show that it matches or outperforms all of the individual scripts that it uses when playing against state-of-the-art bots from the 2014 AIIDE StarCraft competition.

Goal-Directed Design Agents: Integrating Visual Imitation With One-Step Lookahead Optimization for Generative Design

Abstract Engineering design problems often involve large state and action spaces along with highly sparse rewards. Since an exhaustive search of those spaces is not feasible, humans utilize relevant domain knowledge to condense the search space. Deep learning agents (DLAgents) were previously introduced to use visual imitation learning to model design domain knowledge. This note builds on DLAgents and integrates them with one-step lookahead search to develop goal-directed agents capable of enhancing learned strategies for sequentially generating designs. Goal-directed DLAgents can employ human strategies learned from data along with optimizing an objective function. The visual imitation network from DLAgents is composed of a convolutional encoder–decoder network, acting as a rough planning step that is agnostic to feedback. Meanwhile, the lookahead search identifies the fine-tuned design action guided by an objective. These design agents are trained on an unconstrained truss design problem modeled as a sequential, action-based configuration design problem. The agents are then evaluated on two versions of the problem: the original version used for training and an unseen constrained version with an obstructed construction space. The goal-directed agents outperform the human designers used to train the network as well as the previous feedback-agnostic versions of the agent in both scenarios. This illustrates a design agent framework that can efficiently use feedback to not only enhance learned design strategies but also adapt to unseen design problems.

An Energy‐Efficient One‐Shot Scheduling Algorithm for Wireless Sensor Networks

In low‐load wireless sensor networks, the power consumption of the node consists mainly of two parts: data transmission and node state switching. The lower node workload causes low energy consumption on data transmission, and the state switching energy of the node cannot be ignored. This paper proposes a one‐shot time division multiple access (TMDA) scheduling with unlimited channels (SUC) on the assumption that the number of available channels is unlimited. SUC combines the receiver‐based consecutive slot allocation with channel allocation, which minimises the number of node state switching and optimizes energy efficiency. Theoretical analysis demonstrates that the number of channels required by SUC does not exceed , where N indicates the number of nodes. Seeing that the number of available wireless channels is limited in practice, the paper proposes the scheduling with limited channels (SLC) and uses a Lookahead Search mechanism to solve slot conflict. For the scalability of the algorithm, a distributed implementation based on the token change is proposed. The algorithm uses the depth‐first‐search (DFS) to pass the token to all nodes and terminates slot and channel assignment. The simulation results show our algorithm can reduce the energy consumption by minimizing the number of state switching and shorten the data aggregation time by reusing slots among nodes.

A Novel Lookahead Strategy for Delete Relaxation Heuristics in Greedy Best-First Search

Best-first width search (BFWS) is a recent approach to satisficing planning that combines traditional heuristics with novelty measures to achieve a balance between exploration and effective search guidance (exploitation). One such novelty measure is based on counting the number of subgoals achieved on the path from a state in which a relaxed plan was computed. We introduce a new lookahead strategy for greedy best-first search based on this idea, where after each expansion, a bounded lookahead search is guided by relaxed subgoal counting. Furthermore, we combine this technique with partial delete relaxation heuristics to improve the subgoals. Using the hCFF heuristic with online-refinement of conjunctions, we obtain a planner that significantly outperforms the state of the art in satisficing planning on the IPC benchmarks.

A kernel based learning method for non-stationary two-player repeated games

Repeated games is a branch of game theory, where a game can be played several times by the players involved. In this setting, players may not always play the optimal strategy or they may be willing to engage in collaboration or other types of behavior which might lead to a higher long-term profit. Since the same game is repeated for several rounds, and considering a scenario with complete information, it is possible for a player to analyze its opponent’s behavior in order to find patterns. These patterns can then be used to predict the opponent’s actions. Such a setting, where players have mutual information about past moves and do not always play in equilibrium, leads naturally to non-stationary environments, where the players can frequently modify their strategies in order to get ahead in the game. In this work, we propose a novel algorithm based on a string kernel density estimation, which is capable of predicting the opponent’s actions in repeated games and can be used to optimize the player’s profit over time. The prediction is not limited to the next round action. It can also be used to predict a finite sequence of future rounds, which can be combined with a lookahead search scheme with limited depth. In the experiments section, it is shown that the proposed algorithm is able to learn and adapt rapidly, providing good results even if the opponent also adopts an adaptive strategy.

Model Learning for Look-Ahead Exploration in Continuous Control

We propose an exploration method that incorporates lookahead search over basic learnt skills and their dynamics, and use it for reinforcement learning (RL) of manipulation policies. Our skills are multi-goal policies learned in isolation in simpler environments using existing multigoal RL formulations, analogous to options or macroactions. Coarse skill dynamics, i.e., the state transition caused by a (complete) skill execution, are learnt and are unrolled forward during lookahead search. Policy search benefits from temporal abstraction during exploration, though itself operates over low-level primitive actions, and thus the resulting policies does not suffer from suboptimality and inflexibility caused by coarse skill chaining. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parametrized skills as building blocks of the policy itself, as opposed to guiding exploration. We show that the proposed exploration strategy results in effective learning of complex manipulation policies faster than current state-of-the-art RL methods, and converges to better policies than methods that use options or parameterized skills as building blocks of the policy itself, as opposed to guiding exploration.

Point-Based Methodology to Monitor and Control Gene Regulatory Networks via Noisy Measurements

This paper proposes a methodology to monitor and control gene regulatory networks (GRNs) via noisy measurements in an infinite observation space. Toward this end, we employ the partially observed Boolean dynamical system (POBDS) signal model. The proposed methodology consists of offline and online steps. In the offline step, a family of point-based methods is applied to the POBDS model to gather the necessary control policy prior to the online (execution) step. This is accomplished by developing efficient backup and belief expansion processes to make the computation scale with the log of the number of states, as opposed to the complexity of existing point-based methods, which grows with the number of states. In the online step, simultaneous monitoring and control is achieved by a one-step look-ahead search procedure using the optimal state estimation algorithm for the POBDS model, known as the Boolean Kalman filter (BKF), as well as the information gathered in the offline step. The online one-step look-ahead process confers robustness to changes in system dynamics, possibility of starting the execution process before the completion of the offline step. The use of the BKF for simultaneous monitoring and control during the online stage can be key in assessing possible side effects of intervention. The performance of the proposed methodology is investigated through a comprehensive set of numerical experiments using synthetic gene expression data generated from a melanoma GRN.

Game Tree Search Based on Nondeterministic Action Scripts in Real-Time Strategy Games

Significant progress has been made in recent years toward stronger real-time strategy (RTS) game playing agents. Some of the latest approaches have focused on enhancing standard game tree search techniques with a smart sampling of the search space, or on directly reducing this search space. However, experiments have thus far only been performed using small scenarios. We provide experimental results on the performance of these agents on increasingly larger scenarios. Our main contribution is Puppet Search, a new adversarial search framework that reduces the search space by using scripts that can expose choice points to a look-ahead search procedure. Selecting a combination of a script and decisions for its choice points represents an abstract move to be applied next. Such moves can be directly executed in the actual game, or in an abstract representation of the game state, which can be used by an adversarial tree search algorithm. We tested Puppet Search in μRTS, an abstract RTS game popular within the research community, allowing us to directly compare our algorithm against state-of-the-art agents published in the last few years. We show a similar performance to other scripted and search based agents on smaller scenarios, while outperforming them on larger ones.

Sample-Based Tree Search with Fixed and Adaptive State Abstractions

Sample-based tree search (SBTS) is an approach to solving Markov decision problems based on constructing a lookahead search tree using random samples from a generative model of the MDP. It encompasses Monte Carlo tree search (MCTS) algorithms like UCT as well as algorithms such as sparse sampling. SBTS is well-suited to solving MDPs with large state spaces due to the relative insensitivity of SBTS algorithms to the size of the state space. The limiting factor in the performance of SBTS tends to be the exponential dependence of sample complexity on the depth of the search tree. The number of samples required to build a search tree is O((|A|B)^d), where |A| is the number of available actions, B is the number of possible random outcomes of taking an action, and d is the depth of the tree. State abstraction can be used to reduce B by aggregating random outcomes together into abstract states. Recent work has shown that abstract tree search often performs substantially better than tree search conducted in the ground state space. This paper presents a theoretical and empirical evaluation of tree search with both fixed and adaptive state abstractions. We derive a bound on regret due to state abstraction in tree search that decomposes abstraction error into three components arising from properties of the abstraction and the search algorithm. We describe versions of popular SBTS algorithms that use fixed state abstractions, and we introduce the Progressive Abstraction Refinement in Sparse Sampling (PARSS) algorithm, which adapts its abstraction during search. We evaluate PARSS as well as sparse sampling with fixed abstractions on 12 experimental problems, and find that PARSS outperforms search with a fixed abstraction and that search with even highly inaccurate fixed abstractions outperforms search without abstraction. These results establish progressive abstraction refinement as a promising basis for new tree search algorithms, and we propose directions for future work within the progressive refinement framework.

Programmatic Marketing via Reinforcement Learning

This new algorithm which incorporates look-ahead search inside the training loop resulting in rapid improvement and precise and stable learning. By using this new search methods towards programmatic marketing, we massively improve the return of investment on Google Adwords auction by 4.6% in 1 day. We apply simple, gradient-based updates to train the next policy and value network. This appears to be much more stable than incremental, gradient-based policy improvements such as policy gradient or Q-learning that can potentially forget previous improvements.