Mastering the game of Go with deep neural networks and tree search.

Artificial Intelligence and the Common Sense of Animals.

A long-standing goal of artificial intelligence is an algorithm that learns, tabula rasa, superhuman proficiency in challenging domains. Recently, AlphaGo became the first program to defeat a world champion in the game of Go. The tree search in AlphaGo evaluated positions and selected moves using deep neural networks. These neural networks were trained by supervised learning from human expert moves, and by reinforcement learning from self-play. Here we introduce an algorithm based solely on reinforcement learning, without human data, guidance or domain knowledge beyond game rules. AlphaGo becomes its own teacher: a neural network is trained to predict AlphaGo's own move selections and also the winner of AlphaGo's games. This neural network improves the strength of the tree search, resulting in higher quality move selection and stronger self-play in the next iteration. Starting tabula rasa, our new program AlphaGo Zero achieved superhuman performance, winning 100-0 against the previously published, champion-defeating AlphaGo.

With the progress of the times, the field of artificial intelligence (AI) has become one of the hottest fields in the 21st century. Currently, artificial intelligence is successfully used in the retail, financial, and medical industries. Especially in 2016, Google's DeepMind used deep reinforcement learning to train AlphaGo and defeated Lee Sedol, which propelled the field into the public eye. Most people are aware of artificial intelligence, but few understand it. This article will focus on analyzing the literature "Mastering the game of Go with deep neural networks and tree search" and other related articles to introduce the basics of deep reinforcement learning and AlphaGo. Finally, readers will understand how artificial intelligence can successfully imitate humans and defeat humans in Go.

The game of Go is a traditional board game originated in ancient China and has long been viewed as the most challenging of board games for artificial intelligence. In 2016, AlphaGo defeated a human professional player in 19 × 19 game with deep neural networks. The ground-breaking advance in computer Go brought artificial intelligence into public view and raised discussions on how to interact with machines in the future. However, AlphaGo is still a highly intelligent computer program without emotion and personality. This paper presents a novel game design of “Naughty AlphaGo”, which is an emotional Go robot player based on AI algorithm. Result shows that the emotion expression approach based on behavioral and kinematic characteristics can support human players’ perception about the AI player’s emotional states through interaction.

The game of GO has long been regarded as the most challenging game for artificial intelligence because of its enormous search space and the difficulty of evaluating its board positions. In early 2016, the defeat of Lee Sedol by AlphaGo became the milestone of artificial intelligence. AlphaGo’s success lies in that it efficiently combines policy and value networks with Monte Carlo tree search (MCTS). And these deep convolutional neural networks (DCNNs) are trained by the combination of supervised learning and reinforcement learning. However, large convolution operations are computationally-intensive and typically require a powerful computing platform, for example, a graphics processing unit (GPU). Therefore, it is challenging to apply DCCNs in resource-limited embedded systems. Field programmable gate array (FPGA) is proposed to be an appropriate solution to implement real-time DCCNs models. However, the limited bandwidth and on-chip memory storage are the bottlenecks for DCCNs acceleration. In this article, an AlphaGo Policy Network is designed, and efficient hardware architectures are proposed to accelerate the DCCN model. The accelerator can be fit into different FPGAs, providing the balancing between processing speed and hardware resources. As an example, the AlphaGo Policy Network is implemented on Xilinx design suite VCU118, and the results show that our implementation achieved a performance of 3036.32 GOPS and achieved up to 56x speedup compared to CPU and 22.4x speedup compared to GPU.

Monte Carlo Tree Search (MCTS) algorithms show outstanding strengths in decision-making problems such as the game of Go. However, MCTS requires significant computing loads to evaluate many nodes in the decision tree to make a good decision. Parallelizing MCTS node evaluations is challenging because MCTS is a sequential process that each round of tree traversal depends on the previous node evaluations. In this work, we present <italic>SpecMCTS</italic>, a new approach for accelerating MCTS by speculatively traversing the search tree. Many MCTS applications, such as AlphaGo Zero, use a deep neural network (DNN) model to evaluate the tree nodes during the search. SpecMCTS uses a pair of DNN models, the <italic>speculation model</italic> and the <italic>main model</italic>. The faster (but less accurate) speculation model accelerates the sequential tree search while the more accurate main model improves the decision quality. SpecMCTS accelerates MCTS for the game of Go by up to <inline-formula> <tex-math notation="LaTeX">$2.09\times {}$ </tex-math></inline-formula> on the NVIDIA T4 GPU. This performance improvement can be translated into a better decision quality by performing a larger number of tree traversals within the time limit. For a fixed decision time, SpecMCTS shows stronger gameplay (higher win rate) than the original sequential MCTS and state-of-the-art MCTS parallelization approaches.

PurposeThe purpose of this paper is to study the presence of highly cited papers of Nature in social media websites and tools. It also tries to examine the correlation between altmetric and bibliometric indicators.Design/methodology/approachThis descriptive study was carried out using altmetric indicators. The research sample consisted of 1,000 most-cited articles in Nature. In February 2019, the bibliographic information of these articles was extracted from the Scopus database. Then, the titles of all articles were manually searched on Google, and by referring to the article in the journal website and altmetric institution, the data related to social media presence and altmetric score of articles were collected. The data were analyzed using Microsoft Excel and SPSS.FindingsAccording to the results of the study, from 1,000 articles, 989 of them (98.9 per cent) were mentioned at least once in different social media websites and tools. The most used altmetric source in highly cited articles was Mendeley (98.9 per cent), followed by Citeulike (79.8 per cent) and Wikipedia (69.4 per cent). Most Tweets, blog posts, Facebook posts, news stories, readers in Mendeley, Citeulike and Connotea and Wikipedia citations belonged to the article titled “Mastering the game of Go with deep neural networks and tree search”. The highest altmetric score was 3,135 which belonged to this paper. Most tweeters and articles’ readers were from the USA. The membership type of the tweeters was public membership. In terms of fields of study, most readers were PhD students in Agricultural and Biological Sciences. Finally, the results of Spearman’s Correlation revealed positive significant statistical correlation between all altmetric indicators and received citations of highly cited articles (p-value = 0.0001).Practical implicationsThe results of this study can help researchers, editors and editorial boards of journals better understand the importance and benefits of using social media and tools to publish articles.Originality/valueAltmetrics is a relatively new field, and in particular, there are not many studies related to the presence of articles in various social media until now. Accordingly, in this study, a comprehensive altmetric analysis was carried out on 1000 most-cited articles of one of the world's most reliable journals.

Faculty Opinions recommendation of Mastering the game of Go with deep neural networks and tree search.

Gomoku is an ancient board game. The traditional approach to solving the Gomoku is to apply tree search on a Gomoku game tree. Although the rules of Gomoku are straightforward, the game tree complexity is enormous. Unlike other board games such as chess and Shogun, the Gomoku board state is more intuitive. This feature is similar to another famous board game, the game of Go. The success of AlphaGo [5, 6] inspired us to apply a supervised learning method and deep neural network in solving the Gomoku game. We designed a deep convolutional neural network model to help the machine learn from the training data. In our experiment, we got 69% accuracy on the training data and 38% accuracy on the testing data. Finally, we combined the trained deep neural network model with a hard-coded convolution-based Gomoku evaluation function to form a hybrid Gomoku artificial intelligence (AI) which further improved performance.

Recently, theory of collective intelligence (CI) evolution is proposed as a meta algorithm toward artificial general intelligence. But the only implementation of the CI algorithm of the theory is the Monte Carlo tree search (MCTS) used by AlphaZero. Since ant colony optimization (ACO) is an extensively used CI algorithm, it is useful to implement CI evolution using ACO. A genetic version of ACO is adapted to satisfy the CI evolution theory by two methods. One method is realized by using a policy network, namely policy network guided ACO (P-ACO). The other method is realized by using a policy network and a value network, namely policy and value network guided ACO (PV-ACO). Both methods of ACO evolution algorithm are applied to Tic-Tac-Toe and Four in a Row, where traditional ACO played poorly compared to the tree search algorithm, e.g., MCTS. Computational experiments are done to compare both methods with pure ACO and MCTS. As a result, the intelligence level of ACO evolution algorithm quickly exceeds pure ACO and MCTS. In this article, the performance of ACO evolution algorithm is analyzed and the feasibility of applying the CI evolution theory to a specific application is verified.

For the game of Go, Chess, and Shogi (Japanese Chess), deep neural networks (DNNs) have contributed to building accurate evaluation functions, and many studies have attempted to create the so-called value network, which predicts the reward of a given state. A recent study of the value network for the game of Go has shown that a two-headed neural network with two different objectives can be trained effectively and performs better than a single-headed network. One of the two heads is called a value head and the other head, the policy head, predicts the next move at a given state. This multitask training makes the network more robust and improves the generalization performance. In this paper, we show that a simple discriminator network is an alternative target of multitask learning. Compared to the existing deep neural network, our proposed network can be designed more easily because of its simple output. Our experimental results showed that our discriminative target also makes the learning stable and the evaluation function trained by our method is comparable to the training of existing studies in terms of predicting the next move and playing strength.

Deep learning for the game of Go achieved considerable success with the victory of AlphaGo against Ke Jie in May 2017. Thus far, there is no clear understanding of why deep learning performed so well in the game of Go. In this paper, we introduce visualization techniques used in image recognition that provide insights into the function of intermediate layers and the operation of the Go policy network. When used as a diagnostic tool, these visualizations enable us to understand what occurs during the training process of policy networks. Further, we attempt to identify important areas through Grad-CAM and combine it with the Go board to provide explanations for next move decisions.

We present a method of training a reinforcement learning agent to play nine-ball pool. The training process uses a combination of reinforcement learning, deep neural networks and search trees. These technologies have achieved tremendous results in discrete strategy board games, and we extend their applications to pool games, which is a complicated continuous case. Pool types of games have a huge action space, to improve the efficiency of exploration, we use a macro and micro action framework to combine reinforcement learning and the search tree. The agent learns skills such as choosing pockets and control the post-collision position. Our method shows the potential to solve billiards planning problems through AI.

CORR Synthesis: When Should the Orthopaedic Surgeon Use Artificial Intelligence, Machine Learning, and Deep Learning?

Using Deep Learning to Predict Fracture Patterns in Crystalline Solids