Do researchers practice what they preach? An empirical analysis of evaluation criteria

Abstract

In our project, we take part in the debate about evaluation criteria in qualitative management and organisational science (MOS) research. Since the use of quantitative research criteria to evaluate qualitative research, as well as specific theoretically derived qualitative criteria, has been criticised, we look at this topic from an empirical perspective. Based on a comprehensive analysis of 449 articles published from 2011 to 2021 in five top-tier MOS-journals (Academy of Management Journal, Administrative Science Quarterly, Journal of Management, Organization Science, Strategic Management Journal), we analyse how researchers address the issues of rigor and quality. Focusing on the most common evaluation criteria in the articles (validity and trustworthiness), our results indicate that scholars use qualitative evaluation criteria in a pragmatic way. This shows that, in contrast to the epistemological and methodological differences in theoretically derived evaluation criteria, researchers apply similar techniques to ensure validity and trustworthiness. Given that we only had access to published articles demonstrating the state of the art of using evaluation criteria, the current use of evaluation criteria during the research process remains a ‘black box.’ Therefore, to uncover such implicit evaluation criteria, we suggest a follow-up interview study to gain a deeper understanding of this relevant ‘black box’. In doing so, we aim to explore the intentions and experiences of authors who published qualitative research in top-tier journals on the question, “How to deal with evaluation criteria in research practice as well as during the review process?”. As publishing articles in leading journals is very competitive, we argue that these criteria are mainly the outcome of institutional practices, such as the review process and editorial policies. Our follow-up interview study proposes a way to shed light not only on the explicitly mentioned criteria in published articles but also on the underlying implicit criteria in-use during the research process up to publication.