Benchmarking National AI Strategies: Why and How Indicators and Monitoring Can Support Agile Implementation

Abstract

Artificial Intelligence (AI) is considered to be the key technology for the 21st century. Many countries have adopted national AI strategies in order to take advantage of the opportunities of AI and to address important challenges. The German government has also followed suit and formally adopted and published its own national strategy in November 2018. Like many other AI strategies, the German government takes a comprehensive approach, covering implications of AI for research, transfer between research and business development, employment, education, and regulation to name just a few of the most important issue areas. However, the strategy has been criticized for not defining clear and measurable objectives. The lack of concrete goals and clear indicators of success is symptomatic of many strategy papers and announcements in German digital policy. Definitions of clear goals are missing as well as policies to monitor progress and measure success. Politicians and citizens are therefore often left wondering what precisely we are trying to achieve, when it will be achieved, and whether we are making progress. The government’s AI strategy provides the opportunity to do better this time. The advantages are clear. Defining indicators requires the development of cross-departmental goals, which lays the foundation for tracking progress and thus creates the conditions for an effective implementation of the strategy. With this paper, we would like to stimulate a discussion about input and output indicators related to national AI strategies. To generate ideas for the development of such indicators, we examined whether and how already-published AI strategies define their goals and measures to validate the achievement of those goals. The national AI strategies provide some good approaches and ideas, but lack an in-depth and systematic engagement with indicators and benchmarks. In a further chapter, we examine the methodologies of existing AI indices. In both cases, we were concerned with working through core questions and providing an initial overview. We would like to caution the reader that this is not a comprehensive study. But we hope to stimulate further discussion and research with this paper. The majority of indices and reports we examined suffer from significant methodological weaknesses. The reports have generally been received uncritically by the media and the public. Therefore, we also want to spark a critical debate around AI reports and benchmarks. That said, we do not wish to generally call into question the importance and utility of these reports. So, in the third chapter, we set out our own ideas for the development of an empirical foundation for an AI strategy. Our approach hinges on a dynamic interaction with AI trend monitoring, as a means to providing the basis for the continuing engagement with and further development of indicators. With this paper, we hope to contribute to the discussion about how to define goals for national AI strategies and about how to measure them.