Linguistic and semantic characteristics of articles and peer review reports in social and medical sciences

Abstract

Recently, initiatives to share data on peer review (Squazzoni et al, 2020) have brought about better understanding of the peer review process across different disciplines (Buljan et al, 2020, Squazzoni et al 2021a, Squazzoni et al 2021b). Some studies (Strang & Siler, 2015; Siler & Strang, 2016) examined the changes from initial submissions to journals to publication. Another study (Teplitskiy & Bakanic, 2016) explored how well peer reviews predict the article’s citation impact. A study of almost half a million peer review reports from 61 journals (Buljan et al., 2020) showed that peer review reports were longer in social science than in medical journals. It also showed that their language characteristics differed across disciplines (Buljan et al., 2020).The aim of our study was to further examine structural and linguistic differences between articles and the characteristics of the peer review process, including the language used and outcomes of peer review reports, in medicine and health sciences vs. social sciences.To select the articles we used the search engine of the Open Research Central (ORC) platform, which covers different fields, and applied the following filters: “Subject area: Medical and health sciences”, and “Subject area: Social science”. “Article type(s): Research article” filter was also applied. We ended up with 1912 medical and 477 social science articles. To create the samples of articles with clear medical/health vs. social sciences content, we excluded those with a tag for medicine and health sciences and any other disciplinary field except biology and life sciences and those with a tag both for social sciences, and biology and life sciences. This left 408 medical and 54 social science articles. Using the DOIs of filtered articles, we downloaded them manually in an XML format. Articles that had not been reviewed were excluded, yielding a total of 51 articles with a social sciences tag and 361 articles with a medicine and health sciences tag, as well as their peer review texts.We analysed them using the Linguistic Inquiry Word Count (LIWC) text analysis software and word embeddings, a method in which words are given mathematical vector representation. To assess the differences between the articles and their reviews across medical and social sciences, we used the one-way ANOVA and post hoc Tukey’s test. For multivariate frequency distribution of the variables we used the contingency test. All analyses were carried out in JASP, version 0.14.1. Articles from medicine and health sciences and those from social sciences differed in structure. The median word count in the introduction section was 674 (IQR=308.0–637.0) for social sciences and 431 (IQR=420.5–1029.0) for medical sciences. The conclusion section was also longer in social sciences, with 263 words (IQR=135.5–516.0) vs. 94 words (IQR=64.0–168.0) in medical sciences (p<0.01, Mann Whitney). The percentage of articles with merged discussion and conclusion was higher in social sciences, whereas medical articles followed the IMRaD structure more often and contained fewer declarations and non-IMRaD sections. They also contained more figures. Linguistic analysis showed that social science articles had higher word count, higher clout, and a less positive tone.Longer wording was also dominant in peer review reports on social science articles. However, the social and medical sciences did not significantly differ in the characteristics of the peer review process and comments from all stages of review or in the outcome of the peer review process (approved, approved with reservation, or rejected) between the two disciplines. There were also no statistically significant differences in manuscript changes between the disciplines, with the exception of text changes in the introduction section that changed more in the social sciences. The review process for articles in social and medical/health sciences may not differ as much as is usually believed. This, however, may be partly owed to the use of the same ORC platform, which may have uniform policies and processes.