Validation and Evaluation of Predictive Models in Hazard Assessment and Risk Management

Abstract

The paper deals with the validation and evaluation of mathematical models in natural hazard analysis, with a special focus on establishing their predictive power. Although most of the tools and statistics available are common to general classification models, some peculiarities arise in the case of hazard assessment. This is due to the fact that the target for validation, the propensity to develop a dangerous characteristic, is not really known and must be estimated from a (usually) very small sample. This implies that the two types of errors (false positives and false negatives) should be given different meanings. Related to this, a very frequent situation is the presence of prevalence (different proportion of positive and negative cases) in the sample. It is shown that sample prevalence can have a dramatic effect in some very common validation statistics, like the confusion matrix and model efficiency. Here some statistics based on the confusion matrix are presented and discussed, and the use of threshold-independent approaches (especially the ROC plot) is shown. The ROC plot is also proposed as a convenient tool for decision-taking in a risk management context. A general scheme for hazard predictive modeling is finally proposed.