Sloppy Models Research Articles

Sloppy model analysis identifies bifurcation parameters without normal form analysis.

Bifurcation phenomena are common in multidimensional multiparameter dynamical systems. Normal form theory suggests that bifurcations are driven by relatively few combinations of parameters. Models of complex systems, however, rarely appear in normal form, and bifurcations are controlled by nonlinear combinations of the bare parameters of differential equations. Discovering reparameterizations to transform complex equationsinto a normal form is often very difficult, and the reparameterization may not even exist in a closed form. Here we show that information geometry and sloppy model analysis using the Fisher information matrix can be used to identify the combination of parameters that control bifurcations. By considering observations on increasingly long timescales, we find those parameters that rapidly characterize the system's topological inhomogeneities, whether the system is in normal form or not. We anticipate that this novel analytical method, which we call time-widening information geometry (TWIG), will be useful in applied network analysis.

Kinetic Modeling and Parameter Estimation of a Prebiotic Peptide Reaction Network.

Although our understanding of how life emerged on Earth from simple organic precursors is speculative, early precursors likely included amino acids. The polymerization of amino acids into peptides and interactions between peptides are of interest because peptides and proteins participate in complex interaction networks in extant biology. However, peptide reaction networks can be challenging to study because of the potential for multiple species and systems-level interactions between species. We developed and employed a computational network model to describe reactions between amino acids to form di-, tri-, and tetra-peptides. Our experiments were initiated with two of the simplest amino acids, glycine and alanine, mediated by trimetaphosphate-activation and drying to promote peptide bond formation. The parameter estimates for bond formation and hydrolysis reactions in the system were found to be poorly constrained due to a network property known as sloppiness. In a sloppy model, the behavior mostly depends on only a subset of parameter combinations, but there is no straightforward way to determine which parameters should be included or excluded. Despite our inability to determine the exact values of specific kinetic parameters, we could make reasonably accurate predictions of model behavior. In short, our modeling has highlighted challenges and opportunities toward understanding the behaviors of complex prebiotic chemical experiments.

Sloppy Models, Renormalization Group Realism, and the Success of Science

AbstractThe “sloppy models” program originated in systems biology, but has seen applications across a range of fields. Sloppy models are dependent on a large number of parameters, but highly insensitive to the vast majority of parameter combinations. Sloppy models proponents claim that the program may explain the success of science. I argue that the sloppy models program can at best provide a very partial explanation. Drawing a parallel with renormalization group realism, I argue that it would only give us grounds for a minimal kind of scientific realism. Nonetheless, the program can offer certain epistemic virtues.

Corporate Sustainability Development Strategy and Corporate Environmental Governance—The Moderating Role of Corporate Environmental Investments

Environmental degradation and ecological devastation have become widespread global concerns in recent years as a result of the expansion of the international economy. China's rapid economic development has been accompanied by a sloppy economic growth model that has damaged the local ecological environment. The Chinese government intends to improve the ecological environment by the end of 2020 in an effort to direct and improve these environmental issues. The strictest environmental laws became effective in 2015. In light of this, this research uses panel data analysis to examine the environmental strategy and environmental governance of Chinese corporations. This article analyses 14,512 samples of listed mainland Chinese enterprises from 2015 to 2020. This research investigates the connection between Corporate Sustainability Development Strategy and Corporate Environmental Governance, as well as the moderating effect of Corporate Environmental Investments.

Multiple cryoprotectant toxicity model for vitrification solution optimization

Vitrification is a promising cryopreservation technique for complex specimens such as tissues and organs. However, it is challenging to identify mixtures of cryoprotectants (CPAs) that prevent ice formation without exerting excessive toxicity. In this work, we developed a multi-CPA toxicity model that predicts the toxicity kinetics of mixtures containing five of the most common CPAs used in the field (glycerol, dimethyl sulfoxide (DMSO), propylene glycol, ethylene glycol, and formamide). The model accounts for specific toxicity, non-specific toxicity, and interactions between CPAs. The proposed model shows reasonable agreement with training data for single and binary CPA solutions, as well as ternary CPA solution validation data. Sloppy model analysis was used to examine the model parameters that were most important for predictions, providing clues about mechanisms of toxicity. This analysis revealed that the model terms for non-specific toxicity were particularly important, especially the non-specific toxicity of propylene glycol, as well as model terms for specific toxicity of formamide and interactions between formamide and glycerol. To demonstrate the potential for model-based design of vitrification methods, we paired the multi-CPA toxicity model with a published vitrification/devitrification model to identify vitrifiable CPA mixtures that are predicted to have minimal toxicity. The resulting optimized vitrification solution composition was a mixture of 7.4 molal glycerol, 1.4 molal DMSO, and 2.4 molal formamide. This demonstrates the potential for mathematical optimization of vitrification solution composition and sets the stage for future studies to optimize the complete vitrification process, including CPA mixture composition and CPA addition and removal methods.

Nonlinear Reactor Design Optimization With Embedded Microkinetic Model Information

Despite the success of multiscale modeling in science and engineering, embedding molecular-level information into nonlinear reactor design and control optimization problems remains challenging. In this work, we propose a computationally tractable scale-bridging approach that incorporates information from multi-product microkinetic (MK) models with thousands of rates and chemical species into nonlinear reactor design optimization problems. We demonstrate reduced-order kinetic (ROK) modeling approaches for catalytic oligomerization in shale gas processing. We assemble a library of six candidate ROK models based on literature and MK model structure. We find that three metrics—quality of fit (e.g., mean squared logarithmic error), thermodynamic consistency (e.g., low conversion of exothermic reactions at high temperatures), and model identifiability—are all necessary to train and select ROK models. The ROK models that closely mimic the structure of the MK model offer the best compromise to emulate the product distribution. Using the four best ROK models, we optimize the temperature profiles in staged reactors to maximize conversions to heavier oligomerization products. The optimal temperature starts at 630–900K and monotonically decreases to approximately 560 K in the final stage, depending on the choice of ROK model. For all models, staging increases heavier olefin production by 2.5% and there is minimal benefit to more than four stages. The choice of ROK model, i.e., model-form uncertainty, results in a 22% difference in the objective function, which is twice the impact of parametric uncertainty; we demonstrate sequential eigendecomposition of the Fisher information matrix to identify and fix sloppy model parameters, which allows for more reliable estimation of the covariance of the identifiable calibrated model parameters. First-order uncertainty propagation determines this parametric uncertainty induces less than a 10% variability in the reactor optimization objective function. This result highlights the importance of quantifying model-form uncertainty, in addition to parametric uncertainty, in multi-scale reactor and process design and optimization. Moreover, the fast dynamic optimization solution times suggest the ROK strategy is suitable for incorporating molecular information in sequential modular or equation-oriented process simulation and optimization frameworks.

Bayesian, frequentist, and information geometric approaches to parametric uncertainty quantification of classical empirical interatomic potentials

In this paper, we consider the problem of quantifying parametric uncertainty in classical empirical interatomic potentials (IPs) using both Bayesian (Markov Chain Monte Carlo) and frequentist (profile likelihood) methods. We interface these tools with the Open Knowledgebase of Interatomic Models and study three models based on the Lennard-Jones, Morse, and Stillinger-Weber potentials. We confirm that IPs are typically sloppy, i.e., insensitive to coordinated changes in some parameter combinations. Because the inverse problem in such models is ill-conditioned, parameters are unidentifiable. This presents challenges for traditional statistical methods, as we demonstrate and interpret within both Bayesian and frequentist frameworks. We use information geometry to illuminate the underlying cause of this phenomenon and show that IPs have global properties similar to those of sloppy models from fields, such as systems biology, power systems, and critical phenomena. IPs correspond to bounded manifolds with a hierarchy of widths, leading to low effective dimensionality in the model. We show how information geometry can motivate new, natural parameterizations that improve the stability and interpretation of uncertainty quantification analysis and further suggest simplified, less-sloppy models.

Research on the Coupling between the Double Cycle Mode and Technological Innovation Systems: Empirical Evidence from Data Envelopment Analysis and Coupled Coordination

China is striving to build a new dual-circulation development pattern with the aim of leading the coupling of science, technology, and innovation through the active role of financial factors, ultimately promoting the country’s optimal economic development. As a result, this paper constructs a coupled dual-circulation-STI (scientific and technological innovation) relationship model. The domestic and foreign components of the input–output relationship are used to portray the extent of the dual circulation, combined with Index DEA to measure the quality of science and technology innovation under the role of financial factors and the coupled coordination model to analyze the intrinsic correlation between the two. The study used an Index-DEA and value chain analysis model to measure each industry’s technological effectiveness and the degree of internal and external circulation, respectively. The coupled coordination model approach was used to measure the coordination relationship between the double-loop model and technological innovation. Based on the coordination assessment table, the degree of coupling and the natural state of science and technology innovation and dual cycle in each industry were assessed. The study found that the country’s dual-circulation pattern gradually strengthened from 2015–2019, with the financial element also driving an increase in STI. However, due to the twists and turns of the external environment, the coupling power of the dual-circulation STI weakened in some years. On this basis, the paper concludes that the quality of STI, driven by financial factors, has been increasing year on year, and the coupling with dual circulation has been trending upwards. Regarding policy implications, developing countries such as China should continue to invest more financially to create a healthy and sustainable innovation model. Through the innovation-dual-circulation development model, the country should eliminate the sloppy development model in construction and use the power of science and technology to lead social progress.

Bayesian Optimal Experiment Design for Sloppy Systems

In complex dynamical systems, precise and accurate estimation of parameters and quality of predictions depends on the information contained in the experimental data. Choosing experimental schemes that maximize information contained in the data is known as Optimal Experimental Design (OED). Fisher Information Matrix and variance-covariance matrix are the central ideas of OED. However, using OED in a class of models known as sloppy models renders the model less predictive, even though the parameters are estimated with substantial precision. This work introduces a new information gain index as an experiment design criterion in the Bayesian framework. The proposed design criterion is based on what is known as the Bhattacharyya coefficient. Our previous studies show that the information gain index indicates a loss of practical identifiability. Further, it is also an indication of sloppy and stiff parameters. Hence, we extend the information index and its interpretation to joint Gaussian distributions; then, using simulations, we demonstrate that the new experiment design criterion selects experiments that minimize prediction and parameter uncertainty in sloppy models.

Unwinding the model manifold: Choosing similarity measures to remove local minima in sloppy dynamical systems.

In this paper, we consider the problem of parameter sensitivity in models of complex dynamical systems through the lens of information geometry. We calculate the sensitivity of model behavior to variations in parameters. In most cases, models are sloppy, that is, exhibit an exponential hierarchy of parameter sensitivities. We propose a parameter classification scheme based on how the sensitivities scale at long observation times. We show that for oscillatory models, either with a limit cycle or a strange attractor, sensitivities can become arbitrarily large, which implies a high effective dimensionality on the model manifold. Sloppy models with a single fixed point have model manifolds with low effective dimensionality, previously described as a "hyper-ribbon." In contrast, models with high effective dimensionality translate into multimodal fitting problems. We define a measure of curvature on the model manifold which we call the winding frequency that estimates the density of local minima in the model's parameter space. We then show how alternative choices of fitting metrics can "unwind" the model manifold and give low winding frequencies. This prescription translates the model manifold from one of high effective dimensionality into the hyper-ribbon structures observed elsewhere. This translation opens the door for applications of sloppy model analysis and model reduction methods developed for models with low effective dimensionality.

Correspondence between thermodynamics and inference.

We expand upon a natural analogy between Bayesian statistics and statistical physics in which sample size corresponds to inverse temperature. This analogy motivates the definition of two statistical quantities: a learning capacity and a Gibbs entropy. The analysis of the learning capacity, corresponding to the heat capacity in thermal physics, leads to insight into the mechanism of learning and explains why some models have anomalously high learning performance. We explore the properties of the learning capacity in a number of examples, including a sloppy model. Next, we propose that the Gibbs entropy provides a natural device for counting distinguishable distributions in the context of Bayesian inference. We use this device to define a generalized principle of indifference in which every distinguishable model is assigned equal a priori probability. This principle results in a solution to a long-standing problem in Bayesian inference: the definition of an objective or uninformative prior. A key characteristic of this approach is that it can be applied to analyses where the model dimension is unknown and circumvents the automatic rejection of higher-dimensional models in Bayesian inference.

Chebyshev Approximation and the Global Geometry of Model Predictions.

Complex nonlinear models are typically ill conditioned or sloppy; their predictions are significantly affected by only a small subset of parameter combinations, and parameters are difficult to reconstruct from model behavior. Despite forming an important universality class and arising frequently in practice when performing a nonlinear fit to data, formal and systematic explanations of sloppiness are lacking. By unifying geometric interpretations of sloppiness with Chebyshev approximation theory, we rigorously explain sloppiness as a consequence of model smoothness. Our approach results in universal bounds on model predictions for classes of smooth models, capturing global geometric features that are intrinsic to their model manifolds, and characterizing a universality class of models. We illustrate this universality using three models from disparate fields (physics, chemistry, biology): exponential curves, reaction rates from an enzyme-catalyzed chemical reaction, and an epidemiology model of an infected population.

Relative sensitivity analysis of the predictive properties of sloppy models.

Commonly among the model parameters characterizing complex biological systems are those that do not significantly influence the quality of the fit to experimental data, so-called "sloppy" parameters. The sloppiness can be mathematically expressed through saturating response functions (Hill's, sigmoid) thereby embodying biological mechanisms responsible for the system robustness to external perturbations. However, if a sloppy model is used for the prediction of the system behavior at the altered input (e.g. knock out mutations, natural expression variability), it may demonstrate the poor predictive power due to the ambiguity in the parameter estimates. We introduce a method of the predictive power evaluation under the parameter estimation uncertainty, Relative Sensitivity Analysis. The prediction problem is addressed in the context of gene circuit models describing the dynamics of segmentation gene expression in Drosophila embryo. Gene regulation in these models is introduced by a saturating sigmoid function of the concentrations of the regulatory gene products. We show how our approach can be applied to characterize the essential difference between the sensitivity properties of robust and non-robust solutions and select among the existing solutions those providing the correct system behavior at any reasonable input. In general, the method allows to uncover the sources of incorrect predictions and proposes the way to overcome the estimation uncertainties.

The relative merit of empirical priors in non-identifiable and sloppy models: Applications to models of learning and decision-making : Empirical priors.

Formal modeling approaches to cognition provide a principled characterization of observed responses in terms of a set of postulated processes, specifically in terms of parameters that modulate the latter. These model-based characterizations are useful to the extent that there is a clear, one-to-one mapping between parameters and model expectations (identifiability) and that parameters can be recovered from reasonably sized data using a typical experimental design (recoverability). These properties are sometimes not met for certain combinations of model classes and data. One suggestion to improve parameter identifiability and recoverability involves the use of "empirical priors", which constrain parameters according to a previously observed distribution of values. We assessed the efficacy of this proposal using a combination of real and artificial data. Our results showed that a point-estimate variant of the empirical-prior method could not improve parameter recovery systematically. We identified the source of poor parameter recovery in the low information content of the data. As a follow-up step, we developed a fully Bayesian variant of the empirical-prior method and assessed its performance. We find that even such a method that takes the covariance structure of the parameter distributions into account cannot reliably improve parameter recovery. We conclude that researchers should invest additional efforts in improving the informativeness of their experimental designs, as many of the problems associated to impoverished designs cannot be alleviated by modern statistical methods alone.

Inverse Problems in Systems Biology: A Critical Review.

Systems Biology may be assimilated to a symbiotic cyclic interplaying between the forward and inverse problems. Computational models need to be continuously refined through experiments and in turn they help us to make limited experimental resources more efficient. Every time one does an experiment we know that there will be some noise that can disrupt our measurements. Despite the noise certainly is a problem, the inverse problems already involve the inference of missing information, even if the data is entirely reliable. So the addition of a certain limited noise does not fundamentally change the situation but can be used to solve the so-called ill-posed problem, as defined by Hadamard. It can be seen as an extra source of information. Recent studies have shown that complex systems, among others the systems biology, are poorly constrained and ill-conditioned because it is difficult to use experimental data to fully estimate their parameters. For these reasons was born the concept of sloppy models, a sequence of models of increasing complexity that become sloppy in the limit of microscopic accuracy. Furthermore the concept of sloppy models contains also the concept of un-identifiability, because the models are characterized by many parameters that are poorly constrained by experimental data. Then a strategy needs to be designed to infer, analyze, and understand biological systems. The aim of this work is to provide a critical review to the inverse problems in systems biology defining a strategy to determine the minimal set of information needed to overcome the problems arising from dynamic biological models that generally may have many unknown, non-measurable parameters.

Dimension Reduction for Systems with Slow Relaxation

We develop reduced, stochastic models for high dimensional, dissipative dynamical systems that relax very slowly to equilibrium and can encode long term memory. We present a variety of empirical and first principles approaches for model reduction, and build a mathematical framework for analyzing the reduced models. We introduce the notions of universal and asymptotic filters to characterize `optimal' model reductions for sloppy linear models. We illustrate our methods by applying them to the practically important problem of modeling evaporation in oil spills.

Minimization of Emittance at the Cornell Electron Storage Ring With Sloppy Models

Minimization of Emittance at the Cornell Electron Storage Ring With Sloppy Models

The Limitations of Model-Based Experimental Design and Parameter Estimation in Sloppy Systems.

We explore the relationship among experimental design, parameter estimation, and systematic error in sloppy models. We show that the approximate nature of mathematical models poses challenges for experimental design in sloppy models. In many models of complex biological processes it is unknown what are the relevant physical mechanisms that must be included to explain system behaviors. As a consequence, models are often overly complex, with many practically unidentifiable parameters. Furthermore, which mechanisms are relevant/irrelevant vary among experiments. By selecting complementary experiments, experimental design may inadvertently make details that were ommitted from the model become relevant. When this occurs, the model will have a large systematic error and fail to give a good fit to the data. We use a simple hyper-model of model error to quantify a model’s discrepancy and apply it to two models of complex biological processes (EGFR signaling and DNA repair) with optimally selected experiments. We find that although parameters may be accurately estimated, the discrepancy in the model renders it less predictive than it was in the sloppy regime where systematic error is small. We introduce the concept of a sloppy system–a sequence of models of increasing complexity that become sloppy in the limit of microscopic accuracy. We explore the limits of accurate parameter estimation in sloppy systems and argue that identifying underlying mechanisms controlling system behavior is better approached by considering a hierarchy of models of varying detail rather than focusing on parameter estimation in a single model.

A KIM-compliant potfit for fitting sloppy interatomic potentials: application to the EDIP model for silicon

Fitted interatomic potentials are widely used in atomistic simulations thanks to their ability to compute the energy and forces on atoms quickly. However, the simulation results crucially depend on the quality of the potential being used. Force matching is a method aimed at constructing reliable and transferable interatomic potentials by matching the forces computed by the potential as closely as possible, with those obtained from first principles calculations. The potfit program is an implementation of the force-matching method that optimizes the potential parameters using a global minimization algorithm followed by a local minimization polish. We extended potfit in two ways. First, we adapted the code to be compliant with the KIM Application Programming Interface (API) standard (part of the Knowledgebase of Interatomic Models project). This makes it possible to use potfit to fit many KIM potential models, not just those prebuilt into the potfit code. Second, we incorporated the geodesic Levenberg–Marquardt (LM) minimization algorithm into potfit as a new local minimization algorithm. The extended potfit was tested by generating a training set using the KIM environment-dependent interatomic potential (EDIP) model for silicon and using potfit to recover the potential parameters from different initial guesses. The results show that EDIP is a ‘sloppy model’ in the sense that its predictions are insensitive to some of its parameters, which makes fitting more difficult. We find that the geodesic LM algorithm is particularly efficient for this case. The extended potfit code is the first step in developing a KIM-based fitting framework for interatomic potentials for bulk and two-dimensional materials. The code is available for download via https://www.potfit.net.

On the relationship between sloppiness and identifiability

Dynamic models of biochemical networks are often formulated as sets of non-linear ordinary differential equations, whose states are the concentrations or abundances of the network components. They typically have a large number of kinetic parameters, which must be determined by calibrating the model with experimental data. In recent years it has been suggested that dynamic systems biology models are universally sloppy, meaning that the values of some parameters can be perturbed by several orders of magnitude without causing significant changes in the model output. This observation has prompted calls for focusing on model predictions rather than on parameters. In this work we examine the concept of sloppiness, investigating its links with the long-established notions of structural and practical identifiability. By analysing a set of case studies we show that sloppiness is not equivalent to lack of identifiability, and that sloppy models can be identifiable. Thus, using sloppiness to draw conclusions about the possibility of estimating parameter values can be misleading. Instead, structural and practical identifiability analyses are better tools for assessing the confidence in parameter estimates. Furthermore, we show that, when designing new experiments to decrease parametric uncertainty, designs that optimize practical identifiability criteria are more informative than those that minimize sloppiness.