Accurate Uncertainty Estimates Research Articles

UNIQUE: A Framework for Uncertainty Quantification Benchmarking.

Machine learning (ML) models have become key in decision-making for many disciplines, including drug discovery and medicinal chemistry. ML models are generally evaluated prior to their usage in high-stakes decisions, such as compound synthesis or experimental testing. However, no ML model is robust or predictive in all real-world scenarios. Therefore, uncertainty quantification (UQ) in ML predictions has gained importance in recent years. Many investigations have focused on developing methodologies that provide accurate uncertainty estimates for ML-based predictions. Unfortunately, there is no UQ strategy that consistently provides robust estimates about model's applicability on new samples. Depending on the dataset, prediction task, and algorithm, accurate uncertainty estimations might be unfeasible to obtain. Moreover, the optimum UQ metric also varies across applications, and previous investigations have shown a lack of consistency across benchmarks. Herein, the UNIQUE (UNcertaInty QUantification bEnchmarking) framework is introduced to facilitate a comparison of UQ strategies in ML-based predictions. This Python library unifies the benchmarking of multiple UQ metrics, including the calculation of nonstandard UQ metrics (combining information from the dataset and model), and provides a comprehensive evaluation. In this framework, UQ metrics are evaluated for different application scenarios, e.g., eliminating the predictions with the lowest confidence or obtaining a reliable uncertainty estimate for an acquisition function. Taken together, this library will help to standardize UQ investigations and evaluate new methodologies.

ABodyBuilder3: improved and scalable antibody structure predictions.

In this article, we introduce ABodyBuilder3, an improved and scalable antibody structure prediction model based on ABodyBuilder2. We achieve a new state-of-the-art accuracy in the modelling of CDR loops by leveraging language model embeddings, and show how predicted structures can be further improved through careful relaxation strategies. Finally, we incorporate a predicted Local Distance Difference Test into the model output to allow for a more accurate estimation of uncertainties. The software package is available at https://github.com/Exscientia/ABodyBuilder3 with model weights and data at https://zenodo.org/records/11354577.

Uncertainty of Line-of-sight Velocity Measurements of Faint Stars from Low- and Medium-resolution Optical Spectra

Massively multiplexed spectrographs will soon gather large statistical samples of stellar spectra. The accurate estimation of uncertainties on derived parameters, such as the line-of-sight velocity v los, especially for spectra with low signal-to-noise ratios (S/Ns), is paramount. We generated an ensemble of simulated optical spectra of stars as if they were observed with low- and medium-resolution fiber-fed instruments on an 8 m class telescope, similar to the Subaru Prime Focus Spectrograph, and determined v los by fitting stellar templates to the simulated spectra. We compared the empirical errors of the derived parameters—calculated from an ensemble of simulations—to the asymptotic errors determined from the Fisher matrix, as well as from Monte Carlo sampling of the posterior probability. We confirm that the uncertainty of v los scales with the inverse square root of the S/N, but also show how this scaling breaks down at low S/N and analyze the error and bias caused by template mismatch. We outline a computationally optimized algorithm to fit multiexposure data and provide a mathematical model of stellar spectrum fitting that maximizes the so called significance, which allows for calculating the error from the Fisher matrix analytically. We also introduce the effective line count, and provide a scaling relation to estimate the errors of v los measurements based on stellar type. Our analysis covers a range of stellar types with parameters that are typical of the Galactic outer disk and halo, together with analogs of stars in M31 and in satellite dwarf spheroidal galaxies around the Milky Way.

Lithium battery state of health estimation using real-world vehicle data and an interpretable hybrid framework

Accurate point estimation and uncertainty estimation of the state of health (SOH) of battery systems are crucial for alleviating user range anxiety and preventing battery safety incidents. On top of approximately 28 million real-world electric vehicle operation data samples, this paper embeds the categorical boosting algorithm as the base learner of the natural gradient boosting, proposing a novel interpretable N-CatBoost hybrid framework to achieve precise point estimation and uncertainty estimation of the battery SOH. Deep charging segments are selected to calculate the initial capacity, and the nonlinear decay trend of the capacity is derived through the Savitzky-Golay filter. Based on the influencing factors of capacity degradation, health features characterizing battery aging are extracted from electric vehicle data as model inputs. The model's hyperparameters are optimized using the particle swarm optimization algorithm, and it is compared with seven other popular machine learning algorithms. The results indicate that the proposed N-CatBoost model achieves the highest estimation accuracy, with mean absolute percentage error and root mean square error of 0.817 % and 1.156 Ah, respectively. In addition, the Shapley additive explanation method is utilized to make the model interpretable, providing full transparency of all predicted values. More importantly, the developed N-CatBoost model can achieve uncertainty estimation of its predictions, with 100 % of the actual capacity values successfully falling within the model's 99 % prediction interval. Therefore, the proposed N-CatBoost model is a reliable and effective method with great potential for deployment on the cloud side for the SOH estimation of batteries in large-scale vehicles.

Evaluation of Sampling Algorithms Used for Bayesian Uncertainty Quantification of Molecular Dynamics Force Fields.

New Bayesian parameter estimation methods have the capability to enable more physically realistic and reliable molecular dynamics (MD) simulations by providing accurate estimates of uncertainties of force-field (FF) parameters and associated properties. However, the choice of which Bayesian parameter estimation algorithm to use has not been widely investigated, despite its impact on the effective exploration of parameter space. Here, using a case example of the Embedded Atom Method (EAM) FF parameters, we investigated the ramifications of several of the algorithm choices. We found that Ensemble Slice Sampling (ESS) and Affine-Invariant Ensemble Sampling (AIES) demonstrate a new level of superior performance, culminating in more accurate parameter and property estimations with tighter uncertainty bounds, compared to traditional methods such as Metropolis-Hastings (MH), Gradient Search (GS), and Uniform Random Sampler (URS). We demonstrate that Bayesian Uncertainty Quantification with ESS and AIES leads to significantly more accurate and reliable predictions of the FF parameters and properties. The results suggest that ESS and AIES should be used to obtain more accurate parameter and uncertainty estimations while providing deeper physical insights.

Domain knowledge-driven variational recurrent networks for drought monitoring

In the context of climate change, droughts, increasingly frequent and severe, necessitate effective monitoring. Existing methods, such as drought indices and data-driven models, face important limitations. Drought indices are built on prior expert knowledge but lack calibration based on actual drought events, while data-driven models prioritize goodness of fit over real event identification, undermining their credibility and generalization, and also struggling to generalize from regional to large-scale contexts. To address these challenges, here we introduce a hybrid machine learning framework for time series that combines domain knowledge and observational data in a variational recurrent neural network. The network models the joint distribution of total precipitation, air temperature, and real drought events, providing accurate predictions and uncertainty estimates. Extensive experiments focusing on a wide range of European drought events from 2011 to 2018 consistently show that our hybrid model surpasses both drought indices and data-driven models in terms of accuracy in drought detection, underlining its effectiveness, robustness, and stability. Our model achieves the best ROC-AUC (%) results in Afghanistan (79.7 ± 0.5), Italy (84.3 ± 0.6), Russia (89.4 ± 0.2), Europe-0 (84.3 ± 0.1), and Europe-1 (82.8 ± 0.4), effectively capturing the starting and ending times of drought events with lower uncertainty, and also generalizing better for unseen locations.

Performance Comparison and Analysis of UCB, ETC, and Thompson Sampling Algorithms in the Multi-Armed Bandit Problem

In this study, the nuanced interplay between problem complexity, prior knowledge, and the exploration-exploitation balance is examined, given their critical impact on algorithmic performance in diverse settings. To gain a deeper understanding of these dynamics, three algorithms – Upper Confidence Bound (UCB), Explore-Then-Commit (ETC), and Thompson Sampling – are selected for a comprehensive investigation. Each of these algorithms presents unique approaches to handling the challenges posed by varying problem contexts. The UCB algorithm, known for its robustness in balancing exploration and exploitation, is scrutinized for its performance in environments with differing levels of complexity and uncertainty. This algorithm's reliance on confidence bounds makes it particularly relevant in scenarios where accurate estimates of uncertainty can significantly enhance decision-making processes. ETC, on the other hand, is characterized by its phased approach, initially exploring options before committing to a seemingly optimal choice. This study examines how the ETC algorithm's performance is influenced by the availability of prior knowledge and the intricacy of the problem at hand. Its phased nature makes it a subject of interest in environments where initial exploration can yield substantial insights for subsequent exploitation.

Two-stage short-term wind power probabilistic prediction using natural gradient boosting combined with neural network

Accurate wind power probabilistic prediction reflects the uncertainty information of wind power generation, which is the foundation for optimal scheduling of power systems. This study proposes a two-stage probabilistic prediction model combining natural gradient boosting and neural network for accurate uncertainty estimation of short-term output in a wind farm. In the first stage, the selected input features containing historical and future information are fed into a neural network for representation learning. In the second stage, the extracted abstract features are concatenated with the original features, and a natural gradient boosting model is employed to acquire short-term probabilistic forecasts. The experimental results using data from two real wind farms indicate that the proposed hybrid model can generate accurate, sharp, and reliable forecasts. After performing the successive day-ahead prediction task for a month in the first wind farm, the average root mean square error and mean absolute error of the proposed model in the point prediction were 0.1330 and 0.1070, respectively, which were 6.21%–57.29% and 2.96%–62.03% lower than those of comparative models. In addition, the model’s forecasting probability density curves demonstrate high reliability; its coverage probability and the mean width percentage of the interval prediction results under the 90% confidence level were 0.9094 and 0.3696, respectively, which were more suitable than those of five other probabilistic prediction models.

2-D probabilistic inversion of MT data and uncertainty quantification using the Hamiltonian Monte Carlo method

SUMMARY Bayesian methods provide a valuable framework for rigorously quantifying the model uncertainty arising from the inherent non-uniqueness in the magnetotelluric (MT) inversion. However, widely used Markov chain Monte Carlo (MCMC) sampling approaches usually require a significant number of model samples for accurate uncertainty estimates, making their applications computationally challenging for 2-D or 3-D MT problems. In this study, we explore the applicability of the Hamiltonian Monte Carlo (HMC) method for 2-D probabilistic MT inversion. The HMC provides a mechanism for efficient exploration in high-dimensional model space by making use of gradient information of the posterior probability distribution, resulting in a substantial reduction in the number of samples needed for reliable uncertainty quantification compared to the conventional MCMC methods. Numerical examples with synthetic data demonstrate that the HMC method achieves rapid convergence to the posterior probability distribution of model parameters with a limited number of model samples, indicating the computational advantages of the HMC in high-dimensional model space. Finally, we applied the developed approach to the COPROD2 field data set. The statistical models derived from the HMC approach agree well with previous results obtained by 2-D deterministic inversions. Most importantly, the probabilistic inversion provides valuable quantitative model uncertainty information associated with the resistivity structures derived from the observed data, which facilitates model interpretation.

Bayesian variational time-lapse full waveform inversion

SUMMARY Time-lapse seismic full-waveform inversion (FWI) provides estimates of dynamic changes in the Earth’s subsurface by performing multiple seismic surveys at different times. Since FWI problems are highly non-linear and non-unique, it is important to quantify uncertainties in such estimates to allow robust decision making based on the results. Markov chain Monte Carlo (McMC) methods have been used for this purpose, but due to their high computational cost, those studies often require a pre-existing accurate baseline model and estimates of the locations of potential velocity changes, and neglect uncertainty in the baseline velocity model. Such detailed and accurate prior information is not always available in practice. In this study we use an efficient optimization method called stochastic Stein variational gradient descent (sSVGD) to solve time-lapse FWI problems without assuming such prior knowledge, and to estimate uncertainty both in the baseline velocity model and the velocity change over time. We test two Bayesian strategies: separate Bayesian inversions for each seismic survey, and a single joint inversion for baseline and repeat surveys, and compare the methods with standard linearized double difference inversion. The results demonstrate that all three methods can produce accurate velocity change estimates in the case of having fixed (exactly repeatable) acquisition geometries. However, the two Bayesian methods generate significantly more accurate results when acquisition geometries changes between surveys. Furthermore, joint inversion provides the most accurate velocity change and uncertainty estimates in all cases tested. We therefore conclude that Bayesian time-lapse inversion using a joint inversion strategy may be useful to image and monitor subsurface changes, in particular where variations in the results would lead to different consequent decisions.

Normalizing flow based uncertainty estimation for deep regression analysis

Uncertainty estimation is a critical component of building safe and reliable machine learning models. Accurate estimation of uncertainties is essential for identifying and mitigating potential risks and ensuring that machine learning systems operate reliably in real-world scenarios. Various approaches, such as ensemble and Bayesian neural networks have been developed by sampling probability predictions from submodels, which is computationally expensive. At present, these techniques are incapable of precisely delineating the boundary separating in-distribution (ID) and out-of-distribution (OOD) data. To fill up this research gap, this paper presents a normalizing flow based framework to directly predict parameters of prior distributions over the probability with a neural network, the proposed model is able to effectively differentiate between ID and OOD data in regression problems. The posterior distributions learned by the model precisely represent uncertainties for OOD data based solely on ID data, without the need for OOD data during training. This approach has shown promising results in a number of applications, including image depth estimation and image adversarial attacks.

The Galactic bulge exploration

We present a new set of period–absolute magnitude–metallicity (PMZ) relations for single-mode RR Lyrae stars calibrated for the optical GBP, V, G, GRP, near-infrared I, J, H, and Ks passbands. We compiled a large dataset (over 100 objects) of fundamental and first-overtone RR Lyrae pulsators consisting of mean intensity magnitudes, reddenings, pulsation properties, iron abundances, and parallaxes measured by the Gaia astrometric satellite in its third data release. Our newly calibrated PMZ relations encapsulate the most up-to-date ingredients in terms of both data and methodology. They are intended to be used in conjunction with large photometric surveys targeting the Galactic bulge, including the Optical Gravitational Lensing Experiment (OGLE), the Vista Variables in the Vía Láctea Survey (VVV), and the Gaia catalog. In addition, our Bayesian probabilistic approach provides accurate uncertainty estimates of the predicted absolute magnitudes of individual RR Lyrae stars. Our derived PMZ relations provide consistent results when compared to benchmark distances to globular clusters NGC 6121 (also known as M 4), NGC 5139 (also known as omega Cen), and Large and Small Magellanic Clouds, which are stellar systems rich in RR Lyrae stars. Lastly, our Ks-band PMZ relations match well with the previously published PMZ relations based on Gaia data and accurately predict the distance toward the prototype of this class of variables, the eponymic RR Lyr itself.

Bayesian Multistate Bennett Acceptance Ratio Methods.

The multistate Bennett acceptance ratio (MBAR) method is a prevalent approach for computing the free energies of thermodynamic states. In this work, we introduce BayesMBAR, a Bayesian generalization of the MBAR method. By integration of configurations sampled from thermodynamic states with a prior distribution, BayesMBAR computes a posterior distribution of free energies. Using the posterior distribution, we derive free energy estimations and compute their associated uncertainties. Notably, when a uniform prior distribution is used, BayesMBAR recovers the MBAR's result but provides more accurate uncertainty estimates. Additionally, when prior knowledge about free energies is available, BayesMBAR can incorporate this information into the estimation procedure by using nonuniform prior distributions. As an example, we show that by incorporating the prior knowledge about the smoothness of free energy surfaces, BayesMBAR provides more accurate estimates than the MBAR method. Given MBAR's widespread use in free energy calculations, we anticipate BayesMBAR to be an essential tool in various applications of free energy calculations.

Analysis of uncertainty of neural fingerprint-based models.

Machine learning has gained popularity for predicting molecular properties based on molecular structure. This study explores the uncertainty estimates of neural fingerprint-based models by comparing pure graph neural networks (GNN) to classical machine learning algorithms combined with neural fingerprints. We investigate the advantage of extracting the neural fingerprint from the GNN and integrating it into a method known for producing better-calibrated probability estimates. Comparisons are made using three classical machine learning methods and the Chemprop model, considering different molecular representations and calibration techniques. We utilize 19 datasets from Toxcast, reflecting real-world scenarios with balanced accuracies ranging from 0.6 to 0.8. Results demonstrate that neural fingerprints combined with classical machine learning methods exhibit a slight decrease in prediction performance compared to the native Chemprop model. However, these models provide significantly improved uncertainty estimates. Notably, uncertainty estimates of neural fingerprint-based methods remain relatively robust for molecules dissimilar to the training set. This suggests that methods like random forest with neural fingerprints can deliver strong prediction performance and reliable uncertainty estimates. When considering both performance and uncertainty, the calibrated Chemprop model and the combination of neural fingerprints with random forest or support vector classifier (SVC) yield comparable results. Surprisingly, the SVC method shows promising performance when combined with neural or count fingerprints. These findings are particularly relevant in real-world industrial projects where accurate predictions and reliable uncertainty estimates are crucial.

A Variational LSTM Emulator of Sea Level Contribution From the Antarctic Ice Sheet

AbstractThe Antarctic ice sheet (AIS) will be a dominant contributor to global mean sea level rise in the 21st century but remains a major source of uncertainty. The Ice Sheet Model Intercomparison for CMIP6 (ISMIP6) is an ensemble of continental‐scale models for studying the evolution of the AIS and projecting its future contribution to sea level. Due to their complexity and computational cost, ISMIP6 simulations are sparse and generated infrequently. Emulators are smaller‐scale models that approximate ISMs and enable experimentation and exploration into the drivers of sea level change. We introduce a neural network (NN) emulator to approximate the ISMIP6 ensemble, using a variational Long Short‐Term Memory (LSTM) with Monte Carlo dropout to quantify single‐projection uncertainty. The proposed NN emulator is compared to a Gaussian Process (GP) emulator on four criteria: accuracy of point estimates and predictive distributions of individual model projections, approximation of the ensemble projections, and model training time. The NN predicts more accurately on single projections, with a mean absolute error of 0.46 mm Sea Level Equivalent (SLE) versus 0.73 mm SLE for the GP, and has more accurate uncertainty estimates. The NN emulator also better approximates the ensemble distribution of ISMIP6 model projections, with a Kullback‐Leibler divergence of 18.26 versus 199.14 for GP at the projection year 2100. The NN enables more accurate experimentation with a reduced runtime, offering a new tool for understanding the important role of regional precipitation, ice sheet drainage systems, and interannual and longer timescale dynamics.

Exploring multi-b-value diffusion-weighted imaging for more accurate prostate segmentation and uncertainty estimation

Background and objectiveAccurate auto-segmentation on diffusion-weighted imaging (DWI) remains a challenge. A 3D convolutional neural network (CNN) model using DWI was developed to automatically segment the prostate gland and estimate uncertainty in precision treatment planning. MethodsA total of 303 patients who uncervent multi-b-value DWI were retrospectively recruited and divided into training and testing groups based on the examination time. Radiologists labeled the prostate region on DWI images of b = 800 sec/mm2. A 3D U-shaped CNN was trained on multi-b-value DWI images (multi-b-value model) and evaluated using the Dice similarity coefficient (DSC), the Hausdorff distance (HD), and the average surface distance (ASD). Segmentation results were compared to a single-b-value model and registration result from T2-weighted images. The study estimated and analyzed the combination of epistemic and aleatoric uncertainty. ResultsThe DSC, HD, and ASD of our final prediction were 0.928 ± 0.036, 5.472 ± 4.008 mm, and 0.575 ± 0.508 mm. The proposed multi-b-value model achieved a higher DSC on DWI images with all b-values when comparing with the single-b-value model, and superior to the registration result on most of the b-values. The uncertainty helps to effectively filter out prediction errors and indicates that the model is more confident in the middle of the prostate gland, followed by the base and apex of the prostate gland. ConclusionsThe complementary information provided by the multi-b-value DWI images helps to improve both the performance of prostate segmentation, while also providing an estimation of the uncertainty associated with the segmentation prediction, which is clinically useful.

Unified Bayesian network for uncertainty quantification of physiological parameters in dynamic contrast enhanced (DCE) MRI of the liver

Objective. Physiological parameter estimation is affected by intrinsic ambiguity in the data such as noise and model inaccuracies. The aim of this work is to provide a deep learning framework for accurate parameter and uncertainty estimates for DCE-MRI in the liver. Approach. Concentration time curves are simulated to train a Bayesian neural network (BNN). Training of the BNN involves minimization of a loss function that jointly minimizes the aleatoric and epistemic uncertainties. Uncertainty estimation is evaluated for different noise levels and for different out of distribution (OD) cases, i.e. where the data during inference differs strongly to the data during training. The accuracy of parameter estimates are compared to a nonlinear least squares (NLLS) fitting in numerical simulations and in vivo data of a patient suffering from hepatic tumor lesions. Main results. BNN achieved lower root-mean-squared-errors (RMSE) than the NLLS for the simulated data. RMSE of BNN was on overage of all noise levels lower by 33% ± 1.9% for k trans, 22% ± 6% for v e and 89% ± 5% for v p than the NLLS. The aleatoric uncertainties of the parameters increased with increasing noise level, whereas the epistemic uncertainty increased when a BNN was evaluated with OD data. For the in vivo data, more robust parameter estimations were obtained by the BNN than the NLLS fit. In addition, the differences between estimated parameters for healthy and tumor regions-of-interest were significant (p < 0.0001). Significance. The proposed framework allowed for accurate parameter estimates for quantitative DCE-MRI. In addition, the BNN provided uncertainty estimates which highlighted cases of high noise and in which the training data did not match the data during inference. This is important for clinical application because it would indicate cases in which the trained model is inadequate and additional training with an adapted training data set is required.

Learning and optimization under epistemic uncertainty with Bayesian hybrid models

Hybrid (i.e., grey-box) models are a powerful and flexible paradigm for predictive science and engineering. Grey-box models use data-driven constructs to incorporate unknown or computationally intractable phenomena into glass-box mechanistic models. The pioneering work of statisticians Kennedy and O’Hagan introduced a new paradigm to quantify epistemic (i.e., model-form) uncertainty. While popular in several engineering disciplines, prior work using Kennedy–O’Hagan hybrid models focuses on prediction with accurate uncertainty estimates. This work demonstrates computational strategies to deploy Bayesian hybrid models for optimization under uncertainty. Specifically, the posterior distributions of Bayesian hybrid models provide a principled uncertainty set for stochastic programming, chance-constrained optimization, or robust optimization. Through two illustrative case studies, we demonstrate the efficacy of hybrid models, composed of a structurally inadequate glass-box model and Gaussian process bias correction term, for decision-making using limited training data. From these case studies, we develop recommended best practices and explore the trade-offs between different hybrid model architectures.

A spot-specific range uncertainty framework for robust optimization of proton therapy treatments.

An accurate estimation of range uncertainties is essential to exploit the potential of proton therapy. According to Paganetti's study, a value of 2.4% (1.5 standard deviation) is currently recommended for planning robust treatments with Monte Carlo dose engines. This number is based on a dominant contribution from the mean excitation energy of tissues. However, it was recently shown that expressing tissues as a mixture of water and "dry" material in the CT calibration process allowed for a significant reduction of this uncertainty. We thus propose an adapted framework for pencil beam scanning robust optimization. First, we move towards a spot-specific range uncertainty (SSRU) determination. Second, we use the water-based formalism to reduce range uncertainties and, potentially, to spare better the organs atrisk. The stoichiometric calibration was adapted to provide a molecular decomposition (including water) of each voxel of the CT. The SSRU calculation was implemented in MCsquare, a fast Monte Carlo dose engine dedicated to proton therapy. For each spot, a ray-tracing method was used to propagate molecular I-values uncertainties and obtain the corresponding effective range uncertainty. These were then combined with other sources of range uncertainties, according to Paganetti's study of 2012. The method was then assessed on three head-and-neck patients. Two plans were optimized for each patient: the first one with the classical 2.4% flat range uncertainty (FRU), the second one with the variable range uncertainty. Both plans were then compared in terms of target coverage and OAR mean dose reduction. Robustness evaluations were also performed, using the SSRU for both plans in order to simulate errors as realistically aspossible. For patient 1, it was found that the median SSRU was 1.04% (1.5 standard deviation), yielding, therefore, a very large reduction from the 2.4% FRU. All three SSRU plans were found to have a very good robustness level at a 90% confidence interval while sparing OAR better than the classical plan. For instance, in nominal cases, average reductions in the mean dose of 15.7, 8.4, and 13.2% were observed in the left parotid, right parotid, and pharyngeal constrictor muscle, respectively. As expected, the classical plans showed a higher but unnecessary level ofrobustness. Promising results of the SSRU framework were observed on three head-and-neck cases, and more patients should now be considered. The method could also benefit to other tumor sites and, in the long run, the variable part of the range uncertainty could be generalized to other sources of uncertainty in order to move towards more and more patient-specifictreatments.

A Decision‐Relevant Factor‐Fixing Framework: Application to Uncertainty Analysis of a High‐Dimensional Water Quality Model

AbstractFactor Fixing (FF) is a common method for reducing the number of model parameters to lower computational cost. FF typically starts with distinguishing the insensitive parameters from the sensitive and pursues uncertainty quantification (UQ) on the resulting reduced‐order model, fixing each insensitive parameter at a fixed value. There is a need, however, to expand such a common approach to consider the effects of decision choices in the FF‐UQ procedure on metrics of interest. Therefore, to guide the use of FF and increase confidence in the resulting dimension‐reduced model, we propose a new adaptive framework consisting of four principles: (a) re‐parameterize the model first to reduce obvious non‐identifiable parameter combinations, (b) focus on decision relevance especially with respect to errors in quantities of interest (QoI), (c) conduct adaptive evaluation and robustness assessment of errors in the QoI across FF choices as sample size increases, and (d) reconsider whether fixing is warranted. The framework is demonstrated on a spatially‐distributed water quality model. The error in estimates of QoI caused by FF can be estimated using a Polynomial Chaos Expansion (PCE) surrogate model. Built with 70 model runs, the surrogate is computationally inexpensive to evaluate and can provide global sensitivity indices for free. For the selected catchment, just two factors may provide an acceptably accurate estimate of model uncertainty in the average annual load of Total Suspended Solids (TSS), suggesting that reducing the uncertainty in these two parameters is a priority for future work before undertaking further formal uncertainty quantification.