The Evaluation of Artificial Intelligence

Abstract

AI will now proceed along two parallel paths: (1) specialized systems and (2) habile systems, … general, intelligent systems … , having general skill . – Nils J. Nilsson, Eye on the Prize (1995) ARTIFICIAL INTELLIGENCE is a successful discipline. New applications and breakthroughs take us by surprise day after day, fuelled by the increasing power of machine learning. But do AI artefacts of today feature intelligence? If so, how much? Of what kind? How can we know? In this chapter we will see that the field has usually evaluated its artefacts in terms of task performance, not really in terms of intelligence. With that task-oriented perspective, we can say that machines are superhuman at chess, at driving a car and, of course, at solving the Rubik's cube. Beyond these particular tasks, Turing's imitation game, still a touchstone for some, is not used in general practice, except for variants, such as CAPTCHAs. Instead, benchmarks and competitions are the regular way of evaluating AI systems. However, as systems specialise for the new benchmarks, more challenging tasks are being devised. But are these more challenging tasks demanding more intelligence? BARING INTELLIGENCE: THE AI EFFECT The roots of artificial intelligence can be traced back to Ramon Llull's Ars Magna, Thomas Hobbes's Artificial Man , Gottfried Leibniz's Ars Combinatoria and the tradition of building mechanical automata since the middle ages until the early twentieth century. However, it is Alan Turing's celebrated 1950 paper “Can a machine think?” that definitively connects the possibility of an artificial mind with modern computers, as per the notions of a universal Turing machine and the Church-Turing thesis (see Panel 1.2). The name “artificial intelligence” and the official start as a discipline took place during the famous 1956 Dartmouth College's “Summer Research Project on Artificial Intelligence”, which congregated John McCarthy, Marvin Minsky, Nathaniel Rochester, Claude Shannon, Arthur Samuel, Oliver Selfridge, Ray Solomonoff, Allen Newell, Herbert Simon and Trenchard More. An excerpt of the proposal contained the following: “The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. …