Confidence and consistency in discrimination: A new family of evaluation metrics for potential distribution models

Abstract

Potential distribution models (PDMs) are widely applied to understand and predict biogeographic patterns. PDM evaluation, however, presents major challenges, including (1) matches of predictions with observed absences and presences being treated similarly and (2) treatment of predicted presences falling outside the observations as errors, while a major motivation of PDMs is to identify such locations. Our aim was to construct a family of model performance metrics to measure the reliability and transferability of PDMs while providing solutions to the problems mentioned above.Instead of binarisation, predictions are reclassified into three categories for model evaluation: certain negative, uncertain and certain positive predictions. Model performance is tested solely within known presences to reduce the effect of unoccupied but suitable sites registered as observed absences. Metrics were developed for both cases: (1) when the target of the modelling is the identification of potential presence locations and (2) when the target is the evaluation of potential presences and absences equally.The new measures offer evaluation optimised for PDM models. On the one hand, (1) the proposed metrics concentrate on matches within observed presences. Thus, matches within the typically large amount of observed absences do not inflate the metric values. On the other hand, (2) the proposed metrics do not treat all mismatches within observed presences as errors and thus allow exploitation of information in the mismatches, too. Besides the theoretical background, we also provide a new R package for calculating the measures.We tested the metrics both on field and simulation data. Both field-based and simulation-based case studies underlined that the new metrics capture a different aspect of model performance than the traditional metrics, such as AUC, TSS, sensitivity and specificity.We conclude that our metrics help to identify whether the modelling process could capture the preferences of the object well enough to reliably find further suitable sites or stay reliable when transferred in space and time.