The impact of AI—can a robot get into the University of Tokyo?

Abstract

The ‘Todai Robot Project (Can a robot get into the University of Tokyo?)’ was initiated by the National Institute of Informatics (NII), the Japan’s only general academic research institution of informatics, in 2011 as an AI grand challenge. The goal of the project is to create an AI system that answers real questions on university entrance examinations. This paper reports the current status of the project including the underlying technologies we have developed thus far, and the results we obtained from evaluations. There have been various types of AI challenges in the past: the chess and shogi matches against professional players, the quiz show challenge against human champions. Todai Robot Project is unique mainly in the following two senses. First, it targets the integrated intelligent tasks rather than single ones. Examinees must take tests of eight subjects, two social studies, two sciences, Japanese (including Japanese and Chinese classics), English, and two mathematics. The tasks require not only the development of ground-breaking underlying technologies in various AI research areas but also their interdisciplinary research synthesis. Second, it enables us to compare the performance of the software and numerous well-educated students. University entrance examinations in Eastern Asian countries, including Japan, are known to be quite competitive and they cover various skill areas and fields. More than half a million high school graduates take National Center Test for University Admissions (NCTUA), standardized multiple choice style tests, every year in Japan, and less than top 3% students are allowed to take the second written test specially designed to select entrants of theUniversity of Tokyo. Our research team developed a testbed [1, 2] that utilized the resources taken from the history problems asked in NCTUA, and organized international evaluation tasks at NTCIR-9, 10 and 11. The participants pursued mainly two approaches to solving history questions. One is traditional statistical factoid approach. Kanayama and Miyao [3] manually converted original true/false world history questions to a set of factoid-style questions and achieved an accuracy of 65% on the NCTUA data by employing Watson’s factoid QA engine as the backend system. Kano [4] developed domain-independent and language-independent keyword-based system, and achieved an accuracy of 51% on the same data. The other approach is to combine a logical inference engine based on their semantic representation with a statistical classifier. Considering that the accuracy rate of Watson at Jeopardy! Challenge was 69%, it was most likely some deeper language analysis would be required to achieve the accuracy rate of 80%.Tian et al. [5] developed a semantic representation framework that allowed efficient inference while capturing various aspects of natural language semantics. Okita and Liu [6] built a QA system in which they used a semantic parser to acquire meaning representation from history textbooks using common-sense knowledge, and they achieved an accuracy of 68% on the NCTUA data. Higashinaka et al. [7] applieddialogue and machine translation technologies to English exams; taking into account both user intentions and sentiment analysis results yielded significant gains in the dialogue completion task andusing language models constructed from large text corpora led to an accuracy of around 80% in the sentence completion task. It has been a long dream to solve mathematical problems (expressed in natural language and images) by machine since Turing’s age. Though the possibility of complete automatization of mathematics was denied by the undecidable results, a very important theory of mathematics was proved to be decidable: the theory of real closed field (RCF). Furthermore, RCF admits the quantifier elimination. It suggests that the major part of Euclidean geometry, algebra and calculus especially that is asked in the entrance exams is automatizable. We first targeted the problems falling in this category, and developed a prototype system to solve them [8–10].