Revisiting R&I policy assessment in the EU: a semantic analysis of ERAWATCH and RIO reports

Abstract

Abstract The monitoring and evaluation of R&I policies relies on quantitative and qualitative methods with important methodological shortcomings. Quantitative assessments rely on standardized indicators that do not capture context-specific challenges, while qualitative approaches lack comparability. Both approaches are complementary but not always analyzed in combination, thereby considerably limiting their validity. In this study, we propose an alternative approach by using semantic analysis to analyze the large body of evidence available in the 189 annual country-specific qualitative assessments developed between 2011 and 2019 for all EU Member States in the European Commission’s ERAWATCH and RIO reports. We show that the semantic analysis of recurrent qualitative reviews of R&I policies can overcome the limitations of both quantitative scoreboards and ad hoc qualitative reviews, by taking into account country-specific challenges while allowing comparisons across countries. In our analysis, we first identify the main R&I challenges highlighted in all EU countries. Secondly, we focus on a set of four EU Member States to identify structural commonalities and contextual differences across countries. Finally, we focus on the case of Spain to compare the findings from the ERAWATCH and RIO reports with the European Innovation Scoreboard as the main quantitative assessment instrument used in the EU to assess Member States’ R&I performance. By showing that semantic analysis of recurrent and cross-country qualitative assessments of EU R&I policies provide a clear added value to the understanding of the main challenges faced in this field, we recommend the creation of a renewed EU research and innovation observatory.