Model Performance Uncertainty Research Articles

Validating the validation: reanalyzing a large-scale comparison of deep learning and machine learning models for bioactivity prediction

Machine learning methods may have the potential to significantly accelerate drug discovery. However, the increasing rate of new methodological approaches being published in the literature raises the fundamental question of how models should be benchmarked and validated. We reanalyze the data generated by a recently published large-scale comparison of machine learning models for bioactivity prediction and arrive at a somewhat different conclusion. We show that the performance of support vector machines is competitive with that of deep learning methods. Additionally, using a series of numerical experiments, we question the relevance of area under the receiver operating characteristic curve as a metric in virtual screening. We further suggest that area under the precision–recall curve should be used in conjunction with the receiver operating characteristic curve. Our numerical experiments also highlight challenges in estimating the uncertainty in model performance via scaffold-split nested cross validation.

Toward robust early-warning models: a horse race, ensembles and model uncertainty

This paper presents first steps towards robust models for crisis prediction. We conduct a horse race of conventional statistical methods and more recent machine learning methods as early-warning models. As individual models are in the literature most often built in isolation of other methods, the exercise is of high relevance for assessing the relative performance of a wide variety of methods. Further, we test various ensemble approaches to aggregating the information products of the built models, providing more robust basis for measuring country-level vulnerabilities. Finally, we provide approaches to estimating model uncertainty in early-warning exercises, particularly model performance uncertainty and model output uncertainty. The approaches put forward in this paper are shown with Europe as a playground. Generally, our results show that the conventional statistical approaches are outperformed by more advanced machine learning methods, such as k-nearest neighbours and neural networks, and particularly by model aggregation approaches through ensemble learning.

Toward Robust Early-Warning Models: A Horse Race, Ensembles and Model Uncertainty

This paper presents first steps toward robust models for crisis prediction. We conduct a horse race of conventional statistical methods and more recent machine learning methods as early-warning models. As individual models are in the literature most often built in isolation of other methods, the exercise is of high relevance for assessing the relative performance of a wide variety of methods. Further, we test various ensemble approaches to aggregating the information products of the built models, providing a more robust basis for measuring country-level vulnerabilities. Finally, we provide approaches to estimating model uncertainty in early-warning exercises, particularly model performance uncertainty and model output uncertainty. The approaches put forward in this paper are shown with Europe as a playground. Generally, our results show that the conventional statistical approaches are outperformed by more advanced machine learning methods, such as k-nearest neighbors and neural networks, and particularly by model aggregation approaches through ensemble learning.

Toward Robust Early-Warning Models: A Horse Race, Ensembles and Model Uncertainty

This paper presents first steps toward robust models for crisis prediction. We conduct a horse race of conventional statistical methods and more recent machine learning methods as early-warning models. As individual models are in the literature most often built in isolation of other methods, the exercise is of high relevance for assessing the relative performance of a wide variety of methods. Further, we test various ensemble approaches to aggregating the information products of the built models, providing a more robust basis for measuring country-level vulnerabilities. Finally, we provide approaches to estimating model uncertainty in early-warning exercises, particularly model performance uncertainty and model output uncertainty. The approaches put forward in this paper are shown with Europe as a playground. Generally, our results show that the conventional statistical approaches are outperformed by more advanced machine learning methods, such as k-nearest neighbors and neural networks, and particularly by model aggregation approaches through ensemble learning.

Toward Robust Early-Warning Models: A Horse Race, Ensembles and Model Uncertainty

This paper presents first steps toward robust models for crisis prediction. We conduct a horse race of conventional statistical methods and more recent machine learning methods as early-warning models. As individual models are in the literature most often built in isolation of other methods, the exercise is of high relevance for assessing the relative performance of a wide variety of methods. Further, we test various ensemble approaches to aggregating the information products of the built models, providing a more robust basis for measuring country-level vulnerabilities. Finally, we provide approaches to estimating model uncertainty in early-warning exercises, particularly model performance uncertainty and model output uncertainty. The approaches put forward in this paper are shown with Europe as a playground. Generally, our results show that the conventional statistical approaches are outperformed by more advanced machine learning methods, such as k-nearest neighbors and neural networks, and particularly by model aggregation approaches through ensemble learning.

On calibration data selection: The case of stormwater quality regression models

A stormwater quality model should be calibrated and verified against available data before it can be confidently used. This paper mainly examines two questions: how do the size and selection of calibration data sets affect model performances and how should the calibration data sets be selected. Regression models are used to simulate stormwater quality (TSS and COD) with variables characterizing rainfall and flow characteristics. Based on large databases of three catchments in France, several models are calibrated and verified with different data subsets. It is confirmed that the selection of calibration data sets leads to significant uncertainty in model performance. The information content in the calibration data sets is also important in addition to their size. Generally model performances can be improved by using a large size of calibration data sets and by selecting calibration data that are representative of all data. Three methods endeavoring to improve model performance by selecting calibration data either according to model outputs or model inputs are developed based on the principle of choosing calibration data that are representative of the whole data set. The effectiveness of the three selection methods is demonstrated by their application on databases of the three catchments. Model performances can be generally improved by selection methods. The selection methods based on model inputs that consider multi-dimension information perform better than the method with one-dimension information consideration.

Evaluation of past and potential phosphorus uptake at the Orlando Easterly Wetland

The Orlando Easterly Wetland (OEW), located near Christmas, Florida, USA, is among the longer-lived treatment wetlands in the United States. It was established in the late 1980s to reduce nitrogen and phosphorus concentrations from tertiary treated wastewater bound for the St. Johns River. A goal of 0.07 mg/l total phosphorus concentration has been set by the regulating agency (St. Johns River Water Management District). In order to understand and define the operating conditions for which this target could be met, a systematic study of historic phosphorus uptake was performed using a traditional first-order model. Phosphorus uptake performance is shown to correlate well with hydraulic performance for two parallel upstream cells. The first-order model is enhanced with predictive capabilities that acknowledge the correlation between the phosphorus uptake rate constant and the hydraulic loading rate observed in the system. Inherent limitations with the first-order modeling approach are addressed and uncertainty in model performance is used to bound predictions.