A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

Abstract

A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports Darko Hren1, David G. Pina2, Cristopher Norman3, Ana Marušić4 1University of Split Faculty of Humanities and Social Sciences, Split, Croatia 2Research Executive Agency, European Commission, Brussels, Belgium 3LIMSI, CNRS, Université Paris Saclay, Paris, France 4University of Split School of Medicine, Split, Croatia Objectives: To identify themes in the content of reviewers' evaluation reports about proposals submitted to the FP7 Marie Curie Initial Training Networks calls and to explore the content areas in reviewers' comments that differentiate successful from unsuccessful proposals. Methods: We qualitatively analyzed the contents of reviewers' comments to obtain the most common themes within evaluation reports. Given a large amount of textual data, we used a two-step approach. The first step was automated to help reduce the data to the manageable size. We used K-Means clustering to identify the clusters of statements within sections of the evaluation reports. Clusters were visualized in 2-dimensional space using TensorBoard Embedding Projector (https://projector.tensorflow.org/). Principal components analysis (PCA) was used for the visualization of clusters and cosine distances between clusters as a reference to cluster distances. In the second step, the clusters were further grouped into themes using empirical estimates from PCA in step 1 and qualitative analysis based on cluster terms and content. To identify clusters that could potentially be grouped into themes, we used cosine distances lower than 1 as an arbitrary indicator of cluster proximity. Next, we inspected the content of the obtained clusters (terms and example statements) to verify whether and how groups of clusters could be interpreted as broader themes. Inspection of content had precedence over empirical data, i.e. if there was no meaningful overlap in cluster terms and content, even clusters that had cosine distance below 1 were not combined. Finally, we registered the frequencies of obtained themes within each evaluation report (i.e. a number of sentences containing terms belonging to a specific theme) to inspect potential differences between proposals with different decision status. Results: We analyzed the content of reviews of 3666 grant proposals submitted to Marie Curie Initial Training Networks from the FP7 PEOPLE programme in 2008 (n=886), 2010 (n=857), 2011 (n=909), and 2012 (n=1013). Each review had four sections of criteria (C1-Science & Technology Quality, C2-Training, C3-Implementation, and C4-Impact), and each section was divided into two parts (Strengths and Weaknesses). Therefore, 12 separate K-Means cluster analyses were performed in the first step, and we identified between 9 and 12 clusters explaining between 32.7 and 51.6% of the variance. In the second step, we identified between 29 themes within different sections of the evaluation reports (Figure 1A-D). Evaluation reports of proposals from the main and reserve lists included more themes related to strengths and fewer themes related to weaknesses than those of rejected proposals. Themes related to strengths were generally more frequent than those related to weaknesses. Most prominent themes related to strengths were: Research programme and objectives and Multidisciplinary context (C1), Consistency of the research and training programme and Career development (C2), Capacities of research and training (C3), and Outreach activities and Career prospects (C4). Most prominent themes related to weaknesses were: Research programme and objectives and Research methodology (C1), Engagement of ESRs and Training characteristics (C2), Capacities of research and training (C3), and Exploitation and dissemination (C4). Conclusions: We used a novel approach which combined machine learning and qualitative inquiry to identify the most common themes of the evaluation reports of proposals submitted to the FP7 Marie Curie Initial Training Networks calls. The themes reflected general evaluation criteria applied by the European Commission, but also specific themes that distinguished successful from unsuccessful proposals thus demonstrating the validity of the selection process. The obtained results may be used by researchers preparing to apply for grants.