Discovery Precision: An effective metric for evaluating performance of machine learning model for explorative materials discovery

Abstract

The evaluation of machine learning (ML) models in identifying novel materials with superior Figure of Merit (FOM) compared to known materials is of utmost importance for exploring their potential applications. A well-designed evaluation should distinguish between good and poor models, avoid bias in the validation and testing set, and offer straightforward interpretable. To fulfill these requirements, a novel metric called Discovery Precision (DP) was proposed to assess the efficiency of ML models for material discovery in terms of probability. To validate the applicability of DP, we have established a comprehensive testing framework for comparing various validation methods and metrics. Through this framework, the DP metric demonstrates excellent performance in material discovery. Additionally, the simulation results indicate that DP effectively assists in model selection and provides an unbiased estimation of model performance.