Data-driven artificial intelligence to automate researcher assessment

Abstract

This article describes how to utilize data-driven artificial intelligence (AI) to automate researcher assessment using data from profiling systems. We consider that a researcher assessment is done for a purpose and not divorced from a specific target placement. We formulate researcher assessment as a binary classification task, that is, a candidate researcher is classified as either fit or unfit for a given placement. For classifying researchers, we adopt case-based reasoning, a transparent artificial intelligence methodology that implements analogical reasoning, allows adaptation, machine learning, and explainability. This work addresses a human limitation through AI. Given a small number of candidates for a job or award and a clear job description, even if capable of selecting the best fit candidate, human decisions may be neither transparent nor reproducible. The approach in this article describes how to use AI methods to, from a job description, select the best fit candidate while considering career trajectories, providing explanations, and being reproducible. We describe the implementation of the methodology for a hypothetical placement in a real research institute from real but anonymized curriculum vitae from the Brazilian Lattes Database. We describe an experiment demonstrating that the purpose-oriented approach is more accurate than purpose-independent classifiers. The proposed methodology meets various principles from the Leiden Manifesto.