Sensitivity Analysis Of Model Output Research Articles

Modelling rheumatoid arthritis: A hybrid modelling framework to describe pannus formation in a small joint

Rheumatoid arthritis (RA) is a chronic inflammatory disorder that causes pain, swelling and stiffness in the joints, and negatively impacts the life of affected patients. The disease does not have a cure yet, as there are still many aspects of this complex disorder that are not fully understood. While mathematical models can shed light on some of these aspects, to date there are few such models that can be used to better understand the disease. As a first step in the mechanistic understanding of RA, in this study we introduce a new hybrid mathematical modelling framework that describes pannus formation in a small proximal interphalangeal (PIP) joint. We perform numerical simulations with this new model, to investigate the impact of different levels of immune cells (macrophages and fibroblasts) on the degradation of bone and cartilage. Since many model parameters are unknown and cannot be estimated due to a lack of experiments, we also perform a sensitivity analysis of model outputs to various model parameters (single parameters or combinations of parameters). Finally, we discuss how our model could be applied to investigate current treatments for RA, for example, methotrexate, TNF-inhibitors or tocilizumab, which can impact different model parameters.

Extension of the Balbi fire spread model to include the field scale conditions of shrubland fires

The ‘Balbi model’ is a simplified rate of fire spread model aimed at providing computationally fast and accurate simulations of fire propagation that can be used by fire managers under operational conditions. This model describes the steady-state spread rate of surface fires by accounting for both radiation and convection heat transfer processes. In the present work the original Balbi model developed for laboratory conditions is improved by addressing specificities of outdoor fires, such as fuel complexes with a mix of live and dead materials, a larger scale and an open environment. The model is calibrated against a small training dataset (n = 25) of shrubland fires conducted in Turkey. A sensitivity analysis of model output is presented and its predictive capacity against a larger independent dataset of experimental fires in shrubland fuels from different regions of the world (Europe, Australia, New Zealand and South Africa) is tested. A comparison with older versions of the model and a generic empirical model is also conducted with encouraging results. The improved model remains physics-based, faster than real time and fully predictive.

Sensitivity analysis: A discipline coming of age

Sensitivity analysis (SA) as a ‘formal’ and ‘standard’ component of scientific development and policy support is relatively young. Many researchers and practitioners from a wide range of disciplines have contributed to SA over the last three decades, and the SAMO (sensitivity analysis of model output) conferences, since 1995, have been the primary driver of breeding a community culture in this heterogeneous population. Now, SA is evolving into a mature and independent field of science, indeed a discipline with emerging applications extending well into new areas such as data science and machine learning. At this growth stage, the present editorial leads a special issue consisting of one Position Paper on “The future of sensitivity analysis” and 11 research papers on “Sensitivity analysis for environmental modelling” published in Environmental Modelling & Software in 2020–21.

Polynomial chaos expansion for sensitivity analysis of model output with dependent inputs

In this paper, we discuss the sensitivity analysis of model response when the uncertain model inputs are not independent of one other. In this case, two different kinds of sensitivity indices can be evaluated: (i) the sensitivity indices that account for the dependence/correlation of an input or group of inputs with the remainder and (ii) the sensitivity indices that do not account for this dependence. We argue that this distinction applies to any global sensitivity measure. In the present work, we focus on the estimation of variance-based sensitivity indices which are based on the second-order moment of the model response of interest. In particular, we derive new strategies and new computationally efficient methods to assess them, which rely on the polynomial chaos expansion. Several numerical exercises are carried out to demonstrate the performance of the new methods, including a sensitivity analysis of a drainage model posterior to its statistical calibration.

Variance-based sensitivity analysis: The quest for better estimators and designs between explorativity and economy

Variance-based sensitivity indices have established themselves as a reference amongst practitioners of sensitivity analysis of model outputs. A variance-based sensitivity analysis typically produces the first-order sensitivity indices Sj and the so-called total-effect sensitivity indices Tj for the uncertain factors of the mathematical model under analysis.Computational cost is critical in sensitivity analysis. This cost depends upon the number of model evaluations needed to obtain stable and accurate values of the estimates. While efficient estimation procedures are available for Sj (Tarantola et al., 2006), this availability is less the case for Tj (Iooss and Lemaître, 2015). When estimating these indices, one can either use a sample-based approach whose computational cost depends on the number of factors or use approaches based on meta-modelling/emulators (e.g., Gaussian processes).The present work focuses on sample-based estimation procedures for Tj for independent inputs and tests different avenues to achieve an algorithmic improvement over the existing best practices. To improve the exploration of the space of the input factors (design) and the formula to compute the indices (estimator), we propose strategies based on the concepts of economy and explorativity. We then discuss how several existing estimators perform along these characteristics.Numerical results are presented for a set of seven test functions corresponding to different settings (few important factors with low cross-factor interactions, all factors equally important with low cross-factor interactions, and all factors equally important with high cross-factor interactions). We conclude the following from these experiments: a) sample-based approaches based on the use of multiple matrices to enhance the economy are outperformed by designs using fewer matrices but with better explorativity; b) amongst the latter, asymmetric designs perform the best and outperform symmetric designs having corrective terms for spurious correlations; c) improving on the existing best practices is fraught with difficulties; and d) ameliorating the results comes at the cost of introducing extra design parameters.

The Proteus composite index: Towards a better metric for global food security

Abstract This paper proposes a new composite index to measure the multidimensional concept of food security. Although other indicators with the same objective exist, they come with several methodological shortfalls. The Proteus index makes a twofold contribution: a) we apply all the steps needed to tackle the uncertainty sources inherent to building a composite indicator (variable selection, data imputation, normalization, weighting and aggregation) and test the assumptions through a Monte Carlo procedure that applies a variance-based sensitivity analysis of model output; and b) the results are robust over time and are comparable within and between countries, thus allowing a measure that can track the country progress towards food security. We demonstrate that the main sources of output variability are weighting, normalization, data imputation, variable selection, and aggregation in descending order of importance, but interaction effects between the uncertainty sources also play a key role. The index provides a contribution to food security monitoring. While it identifies countries requiring priority attention for their chronic situation, it proves flexible enough to capture sudden onset crises. It also reflects the main drivers that can dramatically affect a country’s food security in the short run, insofar suggesting potential areas of intervention for policy makers.

Symmetrical design for symmetrical global sensitivity analysis of model output

ABSTRACTSymmetrical global sensitivity analysis (SGSA) can help practitioners focusing on the symmetrical terms of inputs whose uncertainties have an impact on the model output, which allows reducing the complexity of the model. However, there remains the challenging problem of finding an efficient method to get symmetrical global sensitivity indices (SGSI) when the functional form of the symmetrical terms is unknown, including numerical and non-parametric situations. In this study, we propose a novel sampling plan, called symmetrical design, for SGSA. As a preliminary experiment for model feature extracting, such plan offers the virtue of run-size economy due to its closure respective to the given group. Using the design, we give estimation methods of SGSI as well as their asymptotic properties respectively for numerical model and non-parametrical model directly by the model outputs, and further propose a significance test for SGSI in non-parametric situation. A case study for a benchmark of GSA and a real data analysis show the effectiveness of the proposed design.

Non-parametric methods for global sensitivity analysis of model output with dependent inputs

This paper addresses the issue of performing global sensitivity analysis of model output with dependent inputs. First, we define variance-based sensitivity indices that allow for distinguishing the independent contributions of the inputs to the response variance from their mutual dependent contributions. Then, two sampling strategies are proposed for their non-parametric, numerical estimation. This approach allows us to estimate the sensitivity indices not only for individual inputs but also for groups of inputs. After testing the accuracy of the non-parametric method on some analytical test functions, the approach is employed to assess the importance of dependent inputs on a computer model for the migration of radioactive substances in the geosphere.

Combining screening and metamodel-based methods: An efficient sequential approach for the sensitivity analysis of model outputs

Sensitivity analysis (SA) is able to identify the most influential parameters of a given model. Application of SA is usually critical for reducing the complexity in the subsequent model calibration and use. Unfortunately it is hardly applied, especially when the model is in the form of a computationally expensive black-box computer program. A possible solution concerns applying SA to the metamodel (i.e., an approximation of the computationally expensive model) instead. Among the other options, the use of Gaussian process metamodels (also known as Kriging metamodels) has been recently proposed for the SA of computationally expensive traffic simulation models. However, the main limitation of this approach is its dependence on the model dimensionality. When the model is high-dimensional, the estimation of the Kriging metamodel may still be problematic due to its high computational cost.In order to overcome this problem, in the present paper, the Kriging-based approach has been combined with the quasi-optimized trajectory based elementary effects (quasi-OTEE) approach for the SA of high-dimensional models. The quasi-OTEE SA is used first to screen the influential and non-influential parameters of a high-dimensional model; then the Kriging-based SA is used to calculate the variance-based sensitivity indices, and to rank the most influential parameters in a more accurate way. The application of the proposed sequential SA is illustrated with several numerical experiments. Results show that the method can properly identify the most influential parameters and their ranks, while the number of model evaluations is considerably less than the variance-based SA (e.g., in one of the tests the sequential SA requires over 50 times less model evaluations than the variance-based SA).

Variance-based sensitivity indices for models with dependent inputs

Computational models are intensively used in engineering for risk analysis or prediction of future outcomes. Uncertainty and sensitivity analyses are of great help in these purposes. Although several methods exist to perform variance-based sensitivity analysis of model output with independent inputs only a few are proposed in the literature in the case of dependent inputs. This is explained by the fact that the theoretical framework for the independent case is set and a univocal set of variance-based sensitivity indices is defined. In the present work, we propose a set of variance-based sensitivity indices to perform sensitivity analysis of models with dependent inputs. These measures allow us to distinguish between the mutual dependent contribution and the independent contribution of an input to the model response variance. Their definition relies on a specific orthogonalisation of the inputs and ANOVA-representations of the model output. In the applications, we show the interest of the new sensitivity indices for model simplification setting.

Variance-based sensitivity analysis of model outputs using surrogate models

If a computer model is run many times with different inputs, the results obtained can often be used to derive a computationally cheaper approximation, or surrogate model, of the original computer code. Thereafter, the surrogate model can be employed to reduce the computational cost of a variance-based sensitivity analysis (VBSA) of the model output. Here, we draw attention to a procedure in which an adaptive sequential design is employed to derive surrogate models and estimate sensitivity indices for different sub-groups of inputs. The results of such group-wise VBSAs are then used to select inputs for a final VBSA. Our procedure is particularly useful when there is little prior knowledge about the response surface and the aim is to explore both the global variability and local nonlinear features of the model output. Our conclusions are based on computer experiments involving the process-based river basin model INCA-N, in which outputs like the average annual riverine load of nitrogen can be regarded as functions of 19 model parameters.

From screening to quantitative sensitivity analysis. A unified approach

The present work is a sequel to a recent one published on this journal where the superiority of ‘radial design’ to compute the ‘total sensitivity index’ was ascertained. Both concepts belong to sensitivity analysis of model output. A radial design is the one whereby starting from a random point in the hyperspace of the input factors one step in turn is taken for each factor. The procedure is iterated a number of times with a different starting random point as to collect a sample of elementary shifts for each factor. The total sensitivity index is a powerful sensitivity measure which can be estimated based on such a sample. Given the similarity between the total sensitivity index and a screening test known as method of the elementary effects (or method of Morris), we test the radial design on this method. Both methods are best practices: the total sensitivity index in the class of the quantitative measures and the elementary effects in that of the screening methods. We find that the radial design is indeed superior even for the computation of the elementary effects method. This opens the door to a sensitivity analysis strategy whereby the analyst can start with a small number of points (screening-wise) and then – depending on the results – possibly increase the numeral of points up to compute a fully quantitative measure. Also of interest to practitioners is that a radial design is nothing else than an iterated ‘One factor At a Time’ (OAT) approach. OAT is a radial design of size one. While OAT is not a good practice, modelers in all domains keep using it for sensitivity analysis for reasons discussed elsewhere (Saltelli and Annoni, 2010) [23]. With the present approach modelers are offered a straightforward and economic upgrade of their OAT which maintain OAT's appeal of having just one factor moved at each step.

Moment Independent Importance Measures: New Results and Analytical Test Cases

Moment independent methods for the sensitivity analysis of model output are attracting growing attention among both academics and practitioners. However, the lack of benchmarks against which to compare numerical strategies forces one to rely on ad hoc experiments in estimating the sensitivity measures. This article introduces a methodology that allows one to obtain moment independent sensitivity measures analytically. We illustrate the procedure by implementing four test cases with different model structures and model input distributions. Numerical experiments are performed at increasing sample size to check convergence of the sensitivity estimates to the analytical values.

Editorial

The present issue contains the abstracts of the works presented at the Sixth International Conference on Sensitivity Analysis of Model output, held in Milan, at Bocconi University from July 19th to July 22nd 2010. The conference has been co-organized by the ELEUSI research center of Bocconi University and by the Joint Research Center of the European Commission.

Evaluation of MARS modeling technique for sensitivity analysis of model output

Sensitivity analysis of computer models can typically require a large number of model runs. When these models are computationally expensive to run, it may be advantageous to invest in computationally cheaper surrogate models (emulators or meta-models) that can provide almost the same output as the original model and estimate the sensitivity indices for each input. In this abstract the MARS method is used to mimic the behavior of a nonlinear and non-additive test function. The results show that, overall, MARS provides acceptable estimates of total sensitivity indices at a much lower cost than using only runs of the original model.

Sensitivity analysis with finite changes: An application to modified EOQ models

In this work, we introduce a new method for the sensitivity analysis of model output in the presence of finite changes in one or more of the exogenous variables. We define sensitivity measures that do not rest on differentiability. We relate the sensitivity measures to classical differential and comparative statics indicators. We prove a result that allows us to obtain the sensitivity measures at the same cost of one-variable-at-a-time methods, thus making their estimation feasible also for computationally intensive models. We discuss in detail the derivation of managerial insights formulating a procedure based on the concept of “Settings”. The method is applied to the sensitivity analysis of a discrete change in optimal order quantity following a jump in the exogenous variables of a non-linear programming inventory model.

Self-validated variance-based methods for sensitivity analysis of model outputs

Global sensitivity analysis (GSA) has the advantage over local sensitivity analysis in that GSA does not require strong model assumptions such as linearity or monotonicity. As a result, GSA methods such as those based on variance decomposition are well-suited to multi-physics models, which are often plagued by large nonlinearities. However, as with many other sampling-based methods, inadequate sample size can badly pollute the result accuracies. A natural remedy is to adaptively increase the sample size until sufficient accuracy is obtained. This paper proposes an iterative methodology comprising mechanisms for guiding sample size selection and self-assessing result accuracy. The elegant features in the proposed methodology are the adaptive refinement strategies for stratified designs. We first apply this iterative methodology to the design of a self-validated first-order sensitivity analysis algorithm. We also extend this methodology to propose a self-validated second-order sensitivity analysis algorithm based on refining replicated orthogonal array designs. Several numerical experiments are given to demonstrate the effectiveness of these methods.

Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index

Abstract Variance based methods have assessed themselves as versatile and effective among the various available techniques for sensitivity analysis of model output. Practitioners can in principle describe the sensitivity pattern of a model Y = f ( X 1 , X 2 , … , X k ) with k uncertain input factors via a full decomposition of the variance V of Y into terms depending on the factors and their interactions. More often practitioners are satisfied with computing just k first order effects and k total effects, the latter describing synthetically interactions among input factors. In sensitivity analysis a key concern is the computational cost of the analysis, defined in terms of number of evaluations of f ( X 1 , X 2 , … , X k ) needed to complete the analysis, as f ( X 1 , X 2 , … , X k ) is often in the form of a numerical model which may take long processing time. While the computational cost is relatively cheap and weakly dependent on k for estimating first order effects, it remains expensive and strictly k-dependent for total effect indices. In the present note we compare existing and new practices for this index and offer recommendations on which to use.

Variance decompositions of nonlinear-dynamic stochastic systems

A variance decomposition approach to quantify the effects of stochastic variables in nonlinear-dynamic models is developed. The decomposition is taken temporally with respect to the source of disturbance. The methodology uses Monte-Carlo methods to estimate the variance decomposition using the ANOVA-like procedures proposed in [G.E.B. Archer, A. Saltelli, I.M. Sobol, Sensitivity measures, anova-like techniques and the use of bootstrap, Journal of Statistical Computation and Simulation 58 (1997) 99–120; A. Saltelli, S. Tarantola, K. Chan, A quantitative model-independent method for global sensitivity analysis of model output, Technometrics 41 (1999) 39–56]. The results in this paper generalize the variance decomposition results that are obtained analytically for linear systems.

Application of global sensitivity analysis of model output to building thermal simulations

This paper deals with a set of applications of global sensitivity analysis to building thermal modelling. The aim is to demonstrate the interest of such tools for model analysis and to encourage their use. Indeed, since the last ten years important improvements have been made in the field of sensitivity analysis and especially in computational methods based on the analyse of variance (ANOVA). Yet, their use in the field of building performance modelling is uncommon. After recalling the concept of ANOVA-based sensitivity analysis and the associated sensitivity indices, we describe some computational methods to compute the global sensitivity indices of model input factors. The emphasis is on the methodology that is computationally cheapest but provides more information. For illustration, they are applied to an actual test-cell thermal model.