What’s the point? The value of mixed methods evaluation in evidencing impact of a scientific research infrastructure

Abstract

In a shift from previous projects, the Europlanet 2024 Research Infrastructure involved an impact evaluation officer, a researcher trained in social sciences and education research, to lead its evaluation efforts. This paper/session presents the findings of the evaluation and, in particular, reflects on the value of including a social sciences perspective. That is, we will highlight not only what the evaluation found in terms of impact in key areas such as scientific and technological impact but will focus on what emerged that might not have been uncovered using our previous evaluation approaches.The purpose of the evaluation was to assess the effectiveness of the RI for its user community. Although the analyses were been broadly structured around the five impact areas identified by the OECD’s Reference Framework for Assessing the Scientific and Socio-Economic Impact of Research Infrastructures (2019), the evaluation also aimed to draw out what contributed to its impact. While evidence has been found for impact in all five impact areas (Scientific, Technological, Training and Education, Economic, Social and Societal), evidence is particularly strong in the first three.Across scientific and technological impact, the Transnational Access (TA) visits – which formed the cornerstone of the RI in many ways – made possible impacts, particularly scientific, that simply would not be possible without the support of the RI, due to the access to facilities, both equipment and expertise, that it provided. These impacts were amplified due to the participation of early career researchers in the programme, ensuring impact into the future. Such impact was only enhanced and strengthened by other areas of scientific impact from Virtual Access activities and Networking Activities(e.g. Expert Exchange programme, Telescope Network) activities.Impact in the area of training and education is also substantial, via a Mentorship programme, Expert Exchange programme, Winter Schools and training workshops. One distinguishing feature of the 2024 RI that the evaluation surfaced was the extensive support for students and early career researchers in particular, and amateur astronomers as well. In addition to the various programmes and workshops, the TA visits made a large contribution to this impact via the participation of early career researchers and students – the visits provided opportunities that are not otherwise available to them, thus accelerating their career development.The Social and Societal impact has also been considerable, albeit more challenging to measure. There has been substantial outreach activity undertaken by TA facilities and TA visitors, and there has also been development of educational materials for school use. Both the management of the RI, as well as leaders of multiple TA facilities have engaged with policymakers and policy initiatives. Impact related to social responsibility is also notable. The mentorship programme and support provided for amateur astronomers widened the reach of those who could participate in and benefit from the activity of the RI. Direct support in the form of conference bursaries and workshops to support applications to the TA programme achieved similar goals. Some of these efforts also contributed to impact in the area of education and training and highlighted the multifaceted nature of the RI’s support for the Europlanet community.The evaluation – particularly more qualitative aspects, such as interviews – reflected that across many of the RI activities, collaboration and networking emerged as key to impact, both as a pathway to impact and an indicator of its attainment. The network element provides support for researchers at all levels and strengthens the community, which in turn contributes to sustainability and improving the science. Put differently, the evaluation identified not only the impact that the RI has achieved but also the way in which it has done so – through its support of the community. In particular, the RI provided access – access to expertise, to equipment (labs, field sites) and to people. This access was supported by a host of interlocking activities that clearly impacts on individuals – from doctoral students to amateur astronomers and beyond – and the science that they engage in, now and in the future.  All of this, critically, does not just lead to good science, but the outward facing nature of much of the RI activity means the impacts will be broader than just on the scientists participating directly and their science, but vitally strengthens the planetary science community – and beyond. By having an evaluator trained in social sciences designing and implementing the evaluation – and bringing that perspective to the data, the evaluation of Europlanet 2024 RI robustly evidenced the way in which scientific impacts were achieved and the vital role of support for the community of planetary scientists in this impact.Europlanet 2024 RI has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 871149.