Game Of Othello Research Articles

Offline Model-Learning by Learning Legal Moves in Othello through Synthetic Negative Reinforcement

One of the biggest dependencies of reinforcement learning is sample efficiency since reinforcement learning agents tend to use an excessively large number of episodes to train. These episodes can be expensive, especially when they are conducted in an online manner. Model-Based Reinforcement Learning (MBRL) and offline learning are two methods to aid this problem; MBRL has been shown to improve sample efficiency while offline learning can reduce the number of online episodes needed by substituting them with less expensive offline ones. However, the use of these two methods together is more challenging, encountering issues such as training off of incomplete distributions. We explore these challenges by testing different combinations of offline and online model-learning through learning the legal moves in the board game Othello. During this process, we encounter the additional challenge of only positive reinforcement where offline episodes only provide information about legal moves. To address this problem, we propose a method named synthetic negative reinforcement which uses pre-existing agent knowledge to make up for the lack of information on illegal moves. Our results demonstrate the efficacy of offline learning using synthetic negative reinforcement on robust distributions of offline data, with agents achieving greater than 97% accuracy predicting the legality of moves. We also demonstrate the evident obstacle that skewed distributions provide to offline model-learning.

Enhancing Mathematical Reasoning in Primary School with the Strategic Board Game Othello

This study explores how the board game Othello can enhance primary school students' mathematical reasoning. Mathematical reasoning is increasingly emphasized in international mathematics curricula, yet both teachers and students face challenges in teaching and learning this important skill. Research shows that strategic games, like Othello, can develop thinking abilities related to mathematical reasoning by providing a context for students to engage in reflective thinking, anticipate future moves, and develop reasoning strategies. However, there is a need to understand how teachers can effectively utilize these games in the classroom to foster mathematical reasoning. We address this, through a design-based research approach consisting of a hermeneutic literature study and the development of design principles for a teaching intervention in a fifth-grade classroom. Data from the intervention is collected through participant observations and group interviews. The findings suggest that Othello can serve as a context for students to engage in mathematical reasoning by making and justifying claims and presenting logical arguments. The study proposes three design principles to scaffold mathematical reasoning during Othello gameplay. These principles focus on introducing and reinforcing the use of the "if...then" formulation, promoting exploratory talk, encouraging reflection on strategies, and fostering collaborative reasoning. The results indicate that the design principles positively impacted students' ability to reason mathematically. This paper contributes to the field of mathematics education and game-based learning by providing a practice-oriented perspective on designing mathematical instruction for reasoning using a specific board game in a primary school setting. The findings offer insights into the potential of strategic board games like Othello to enhance students' mathematical reasoning skills. The design principles proposed in this study can guide teachers in developing effective instructional approaches to support students' mathematical reasoning development.

Olivaw: Mastering Othello Without Human Knowledge, nor a Fortune

We introduce OLIVAW, an AI Othello player adopting the design principles of the famous AlphaGo programs. The main motivation behind OLIVAW was to attain exceptional competence in a non-trivial board game at a tiny fraction of the cost of its illustrious predecessors. In this paper, we show how the AlphaGo Zero's paradigm can be successfully applied to the popular game of Othello using only commodity hardware and free cloud services. While being simpler than Chess or Go, Othello maintains a considerable search space and difficulty in evaluating board positions. To achieve this result, OLIVAW implements some improvements inspired by recent works to accelerate the standard AlphaGo Zero learning process. The main modification implies doubling the positions collected per game during the training phase, by including also positions not played but largely explored by the agent. We tested the strength of OLIVAW in three different ways: by pitting it against Edax, the strongest open-source Othello engine, by playing anonymous games on the web platform OthelloQuest, and finally in two in-person matches against top-notch human players: a national champion and a former world champion.

Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples

In this article, we propose a novel algorithm for deep reinforcement learning named Expert Q-learning. Expert Q-learning is inspired by Dueling Q-learning and aims to incorporate semi-supervised learning into reinforcement learning through splitting Q-values into state values and action advantages. We require that an offline expert assesses the value of a state in a coarse manner using three discrete values. An expert network is designed in addition to the Q-network, which updates each time following the regular offline minibatch update whenever the expert example buffer is not empty. Using the board game Othello, we compare our algorithm with the baseline Q-learning algorithm, which is a combination of Double Q-learning and Dueling Q-learning. Our results show that Expert Q-learning is indeed useful and more resistant to the overestimation bias. The baseline Q-learning algorithm exhibits unstable and suboptimal behavior in non-deterministic settings, whereas Expert Q-learning demonstrates more robust performance with higher scores, illustrating that our algorithm is indeed suitable to integrate state values from expert examples into Q-learning.

Characterizing the Nature of Probability-Based Proof Number Search: A Case Study in the Othello and Connect Four Games

Variants of best-first search algorithms and their expansions have continuously been introduced to solve challenging problems. The probability-based proof number search (PPNS) is a best-first search algorithm that can be used to solve positions in AND/OR game tree structures. It combines information from explored (based on winning status) and unexplored (through Monte Carlo simulation) nodes from a game tree using an indicator called the probability-based proof number (PPN). In this study, PPNS is employed to solve randomly generated positions in Connect Four and Othello, in which the results are compared with the two well-known best-first search algorithms (proof number search (PNS) and Monte Carlo proof number search). Adopting a simple improvement parameter in PPNS reduces the number of nodes that need to be explored by up to 57%. Moreover, further observation showed the varying importance of information from explored and unexplored nodes in which PPNS relies critically on the combination of such information in earlier stages of the Othello game. Discussion and insights from these findings are provided where the potential future works are briefly described.

CNN 기반 몬테카를로 트리 탐색 및 강화학습을 이용한 인공지능 오델로 게임 에이전트

본 논문에서는 정책과 가치가 단일 신경망에 의해 표현되고 학습되는 구조의 신경망을 기반으로 하여 몬테카를로 트리탐색을 적용한 인공지능 오델로 게임 에이전트의 구현을 제안하였다. 본 논문에서 적용하는 신경망 학습은 인공지능플레이어의 자가 대국을 통해 생성된 학습 데이터를 이용하여 수행되는데 기존의 정책이 MCTS를 통해 얻은 강력한 정책을 따라가도록 CNN을 학습한다. 본 논문에서 제안한 인공지능 오델로의 성능 평가를 위해 MPC 탐색을 사용하는 현존 최강 오델로 프로그램인 Wzebra 및 Tothello와 대국을 통해 성능을 비교하였고, 최소최대탐색 알고리즘이 적용된 기존의 에이전트와 본 논문에서 제안하는 에이전트의 신경망 학습 진행에 따른 중간 단계별 에이전트와의 대국을 통해 학습 진행에 따른 승률 변화를 관찰하였다.

Performance comparison of different momentum techniques on deep reinforcement learning

ABSTRACTRecently, the popularity of deep artificial neural networks has increased considerably. Generally, the method used in the training of these structures is simple gradient descent. However, training a deep structure with simple gradient descent can take quite a long time. Some additional approaches have been utilized to solve this problem. One of these techniques is the momentum that accelerates gradient descent learning. Momentum techniques can be used for supervised learning as well as for reinforcement learning. However, its efficiency may vary due to the dissimilarities in two learning processes. While the expected values of inputs are clearly known in supervised learning, it may take long-running iterations to reach the exact expected values of the states in reinforcement learning. In an online learning approach, a deep neural network should not memorize and continue to converge with the more precise values that exist over time during these iterations. For this reason, it is necessary to use a momentum technique that both adapt to reinforcement learning and accelerate the learning process. In this paper, the performance of different momentum techniques is compared with the Othello game benchmark. Test results show that the Nesterov momentum technique provided a more effective generalization with an online reinforcement learning approach.

An Implementation of Othello Game Player Using ANN based Records Learning and Minimax Search Algorithm

An Implementation of Othello Game Player Using ANN based Records Learning and Minimax Search Algorithm

Coevolutionary CMA-ES for Knowledge-Free Learning of Game Position Evaluation

One weakness of coevolutionary algorithms observed in knowledge-free learning of strategies for adversarial games has been their poor scalability with respect to the number of parameters to learn. In this paper, we investigate to what extent this problem can be mitigated by using Covariance Matrix Adaptation Evolution Strategy, a powerful continuous optimization algorithm. In particular, we employ this algorithm in a competitive coevolutionary setup, denoting this setting as Co-CMA-ES. We apply it to learn position evaluation functions for the game of Othello and find out that, in contrast to plain (co)evolution strategies, Co-CMA-ES learns faster, finds superior game-playing strategies and scales better. Its advantages come out into the open especially for large parameter spaces of tens of hundreds of dimensions. For Othello, combining Co-CMA-ES with experimentally-tuned derandomized systematic $n$ -tuple networks significantly improved the current state of the art. Our best strategy outperforms all the other Othello 1-ply players published to date by a large margin regardless of whether the round-robin tournament among them involves a fixed set of initial positions or the standard initial position but randomized opponents. These results show a large potential of CMA-ES-driven coevolution, which could be, presumably, exploited also in other games.

Preference Learning for Move Prediction and Evaluation Function Approximation in Othello

This paper investigates the use of preference learning as an approach to move prediction and evaluation function ap- proximation, using the game of Othello as a test domain. Using the same sets of features, we compare our approach with least squares temporal difference learning, direct classification, and with the Bradley-Terry model, fitted using minorization-maximization (MM). The results show that the exact way in which preference learning is applied is critical to achieving high performance. Best results were obtained using a combination of board inversion and pair-wise preference learning. This combination significantly out- performed the others under test, both in terms of move prediction accuracy, and in the level of play achieved when using the learned evaluation function as a move selector during game play.

IMPACT ANALYSIS OF THE CAPABILITY ON STRATEGY VIA OTHELLO GAME

Othello is a game that is already antiquated, but the game is still played by many people. Othello game is played by 2 people. If you want to win over the opponent, you must have a good strategy so that all the pieces on the board into the color of the chip.Each player would continue to seek a new strategy to make the pieces multiply. Under these conditions, will result in job terstimulasinya brain to strategize. This of course has an impact on the development of creativity in the organization's strategy.This study was conducted to prove that the game Othello is able to train its player in developing strategies. The results showed that in the Othello game, players are able to train level strategy if he plays continuously. Automatically, playing the game of Othello can stimulate the brain so as to improve the ability to strategize.Keyword Strategic,thinking, critical thinking, tactic

On Scalability, Generalization, and Hybridization of Coevolutionary Learning: A Case Study for Othello

This study investigates different methods of learning to play the game of Othello. The main questions posed concern scalability of algorithms with respect to the search space size and their capability to generalize and produce players that fare well against various opponents. The considered algorithms represent strategies as n-tuple networks, and employ self-play temporal difference learning (TDL), evolutionary learning (EL) and coevolutionary learning (CEL), and hybrids thereof. To assess the performance, three different measures are used: score against an a priori given opponent (a fixed heuristic strategy), against opponents trained by other methods (round-robin tournament), and against the top-ranked players from the online Othello League. We demonstrate that although evolutionary-based methods yield players that fare best against a fixed heuristic player, it is the coevolutionary temporal difference learning (CTDL), a hybrid of coevolution and TDL, that generalizes better and proves superior when confronted with a pool of previously unseen opponents. Moreover, CTDL scales well with the size of representation, attaining better results for larger n-tuple networks. By showing that a strategy learned in this way wins against the top entries from the Othello League, we conclude that it is one of the best 1-ply Othello players obtained to date without explicit use of human knowledge.

Neural-fitted TD-leaf learning for playing Othello with structured neural networks.

This paper describes a methodology for quickly learning to play games at a strong level. The methodology consists of a novel combination of three techniques, and a variety of experiments on the game of Othello demonstrates their usefulness. First, structures or topologies in neural network connectivity patterns are used to decrease the number of learning parameters and to deal more effectively with the structural credit assignment problem, which is to change individual network weights based on the obtained feedback. Furthermore, the structured neural networks are trained with the novel neural-fitted temporal difference (TD) learning algorithm to create a system that can exploit most of the training experiences and enhance learning speed and performance. Finally, we use the neural-fitted TD-leaf algorithm to learn more effectively when look-ahead search is performed by the game-playing program. Our extensive experimental study clearly indicates that the proposed method outperforms linear networks and fully connected neural networks or evaluation functions evolved with evolutionary algorithms.

Improving Generalization Performance in Co-Evolutionary Learning

Recently, the generalization framework in co-evolutionary learning has been theoretically formulated and demonstrated in the context of game-playing. Generalization performance of a strategy (solution) is estimated using a collection of random test strategies (test cases) by taking the average game outcomes, with confidence bounds provided by Chebyshev's theorem. Chebyshev's bounds have the advantage that they hold for any distribution of game outcomes. However, such a distribution-free framework leads to unnecessarily loose confidence bounds. In this paper, we have taken advantage of the near-Gaussian nature of average game outcomes and provided tighter bounds based on parametric testing. This enables us to use small samples of test strategies to guide and improve the co-evolutionary search. We demonstrate our approach in a series of empirical studies involving the iterated prisoner's dilemma (IPD) and the more complex Othello game in a competitive co-evolutionary learning setting. The new approach is shown to improve on the classical co-evolutionary learning in that we obtain increasingly higher generalization performance using relatively small samples of test strategies. This is achieved without large performance fluctuations typical of the classical approach. The new approach also leads to faster co-evolutionary search where we can strictly control the condition (sample sizes) under which the speedup is achieved (not at the cost of weakening precision in the estimates).

사례 기반 추론법을 이용한 오델로 게임 개발에 관한 연구

AI(Artificial Intelligence) refers to the area of computer engineering and IT technology that focuses on the methodology and creation of intelligent agents. The Othello game is often produced with AI, since it is played with relatively simple rules on a board and on a limited space of 8 rows and 8 columns. Previous algorithms take longer time than desirable and often fail to face new circumstances, as they search for all the possible cases and rules. In order to solve this crucial weakness, we propose that a CBR algorithm be applied to Orthello. Case-Based Reasoning(CBR), is the process of solving new problems based on the solutions of the past similar problems. We can apply this process to Othello and expedite the process of computer reasoning for a solution to new cases based on the data from accumulated past cases. Then, these new solutions are dynamically added to the set of past cases so that it becomes harder for players(users) to be able to read the pattern. The proposed system in which a CBR algorithm is applied to the Othello game makes the computation process faster and the game harder to play.

A Study of Artificial Neural Network Architectures for Othello Evaluation Functions

In this study, we use temporal difference learning (TDL) to investigate the ability of 20 different artificial neural network (ANN) architectures to learn othello game board evaluation functions. The ANN evaluation functions are applied to create a strong othello player using only 1-ply search. In addition to comparing many of the ANN architectures seen in the literature, we introduce several new architectures that consider the game board symmetry. Both embedding the game board symmetry into the network architecture through weight sharing and the outright removal of symmetry through symmetry removal are explored. Experiments varying the number of inputs per game board square from one to three, the number of hidden nodes, and number of hidden layers are also performed. We found it advantageous to consider game board symmetry in the form of symmetry by weight sharing; and that an input encoding of three inputs per square outperformed the one input per square encoding that is commonly seen in the literature. Furthermore, architectures with only one hidden layer were strongly outperformed by architectures with multiple hidden layers. A standard weighted-square board heuristic evaluation function from the literature was used to evaluate the quality of the trained ANN othello players. One of the ANN architectures introduced in this study, an ANN implementing weight sharing and consisting of three hidden layers, using only a 1-ply search, outperformed a weighted-square test heuristic player using a 6-ply minimax search.

Application of reinforcement learning to the game of Othello

Operations research and management science are often confronted with sequential decision making problems with large state spaces. Standard methods that are used for solving such complex problems are associated with some difficulties. As we discuss in this article, these methods are plagued by the so-called curse of dimensionality and the curse of modelling. In this article, we discuss reinforcement learning, a machine learning technique for solving sequential decision making problems with large state spaces. We describe how reinforcement learning can be combined with a function approximation method to avoid both the curse of dimensionality and the curse of modelling. To illustrate the usefulness of this approach, we apply it to a problem with a huge state space—learning to play the game of Othello. We describe experiments in which reinforcement learning agents learn to play the game of Othello without the use of any knowledge provided by human experts. It turns out that the reinforcement learning agents learn to play the game of Othello better than players that use basic strategies.

Observing the Evolution of Neural Networks Learning to Play the Game of Othello

A study was conducted to find out how game-playing strategies for Othello (also known as reversi) can be learned without expert knowledge. The approach used the coevolution of a fixed-architecture neural-network-based evaluation function combined with a standard minimax search algorithm. Comparisons between evolving neural networks and computer players that used deterministic strategies allowed evolution to be observed in real-time. Neural networks evolved to outperform the computer players playing at higher ply-depths, despite being handicapped by playing black and using minimax at ply-depth of two. In addition, the playing ability of the population progressed from novice, to intermediate, and then to master's level. Individual neural networks discovered various game-playing strategies, starting with positional and later mobility. These results show that neural networks can be evolved as evaluation functions, despite the general difficulties associated with this approach. Success in this case was due to a simple spatial preprocessing layer in the neural network that captured spatial information, self-adaptation of every weight and bias of the neural network, and a selection method that allowed a diverse population of neural networks to be carried forward from one generation to the next.

Parallel Randomized Best-First Minimax Search

We describe a novel parallel randomized search algorithm for two-player games. The algorithm is a randomized version of Korf and Chickering's best-first search. Randomization both fixes a defect in the original algorithm and introduces significant parallelism. An experimental evaluation demonstrates that the algorithm is efficient (in terms of the number of search-tree vertices that it visits) and highly parallel. On incremental random game trees the algorithm outperforms Alpha-Beta, and speeds up by up to a factor of 18 (using 35 processors). In comparison, Jamboree [ICCA J. 18 (1) (1995) 3–19] speeds up by only a factor of 6. The algorithm outperforms Alpha-Beta in the game of Othello. We have also evaluated the algorithm in a Chess-playing program using the board-evaluation code from an existing Alpha-Beta-based program (Crafty). On a single processor our program is slower than Crafty by about a factor of 7, but with multiple processors it outperforms it: with 64 processors our program is always faster, usually by a factor of 5, sometimes much more.

Improvements of the Penalty Avoiding Rational Policy Making Algorithm and an Application to the Othello Game

The purpose of reinforcement learning is to learn an optimal policy in general. However, in 2-players games such as the othello game, it is important to acquire a penalty avoiding policy. In this paper, we focus on formation of a penalty avoiding policy based on the Penalty Avoiding Rational Policy Making algorithm [Miyazaki 01]. In applying it to large-scale problems, we are confronted with the curse of dimensionality. We introduce several ideas and heuristics to overcome the combinational explosion in large-scale problems. First, we propose an algorithm to save the memory by calculation of state transition. Second, we describe how to restrict exploration by two type knowledge; KIFU database and evaluation funcion. We show that our learning player can always defeat against the well-known othello game program KITTY.