Mean Objective Cost Of Uncertainty Research Articles

Neural message-passing for objective-based uncertainty quantification and optimal experimental design

Various real-world scientific applications involve the mathematical modeling of complex uncertain systems with numerous unknown parameters. Accurate parameter estimation is often practically infeasible in such systems, as the available training data may be insufficient and the cost of acquiring additional data may be high. In such cases, based on a Bayesian paradigm, we can design robust operators retaining the best overall performance across all possible models and design optimal experiments that can effectively reduce uncertainty to enhance the performance of such operators maximally. While objective-based uncertainty quantification (objective-UQ) based on MOCU (mean objective cost of uncertainty) provides an effective means for quantifying uncertainty in complex systems, the high computational cost of estimating MOCU has been a challenge in applying it to real-world scientific/engineering problems. In this work, we propose a novel scheme to reduce the computational cost for objective-UQ via MOCU based on a data-driven approach. We adopt a neural message-passing model for surrogate modeling, incorporating a novel axiomatic constraint loss that penalizes an increase in the estimated system uncertainty. As an illustrative example, we consider the optimal experimental design (OED) problem for uncertain Kuramoto models, where the goal is to predict the experiments that can most effectively enhance robust synchronization performance through uncertainty reduction. We show that our proposed approach can accelerate MOCU-based OED by four to five orders of magnitude, without any visible performance loss compared to the state-of-the-art. The proposed approach applies to general OED tasks, beyond the Kuramoto model.

Graph-Based Bayesian Optimization for Large-Scale Objective-Based Experimental Design

Design is an inseparable part of most scientific and engineering tasks, including real and simulation-based experimental design processes and parameter/hyperparameter tuning/optimization. Several model-based experimental design techniques have been developed for design in domains with partial available knowledge about the underlying process. This article focuses on a powerful class of model-based experimental design called the mean objective cost of uncertainty (MOCU). The MOCU-based techniques are objective-based, meaning that they take the main objective of the process into account during the experimental design process. However, the lack of scalability of MOCU-based techniques prevents their application to most practical problems, including large discrete or combinatorial spaces. To achieve a scalable objective-based experimental design, this article proposes a graph-based MOCU-based Bayesian optimization framework. The correlations among samples in the large design space are accounted for using a graph-based Gaussian process, and an efficient closed-form sequential selection is achieved through the well-known expected improvement policy. The proposed framework's performance is assessed through the structural intervention in gene regulatory networks, aiming to make the network away from the states associated with cancer.

Accelerating Optimal Experimental Design for Robust Synchronization of Uncertain Kuramoto Oscillator Model Using Machine Learning

The need for efficient computational screening of molecular candidates that possess desired properties frequently arises in various scientific and engineering problems, including drug discovery and materials design. However, the large size of the search space containing the candidates and the substantial computational cost of high-fidelity property prediction models makes screening practically challenging. In this work, we propose a general framework for constructing and optimizing a virtual screening (HTVS) pipeline that consists of multi-fidelity models. The central idea is to optimally allocate the computational resources to models with varying costs and accuracy to optimize the return-on-computational-investment (ROCI). Based on both simulated as well as real data, we demonstrate that the proposed optimal HTVS framework can significantly accelerate screening virtually without any degradation in terms of accuracy. Furthermore, it enables an adaptive operational strategy for HTVS, where one can trade accuracy for efficiency.

Experimental Design via Generalized Mean Objective Cost of Uncertainty

The mean objective cost of uncertainty (MOCU) quantifies the performance cost of using an operator that is optimal across an uncertainty class of systems as opposed to using an operator that is optimal for a particular system. MOCU-based experimental design selects an experiment to maximally reduce MOCU, thereby gaining the greatest reduction of uncertainty impacting the operational objective. The original formulation applies to finding optimal system operators, where optimality is with respect to a cost function, such as mean-square error; and the prior distribution governing the uncertainty class relates directly to the underlying physical system. Here, we provide a generalized MOCU and the corresponding experimental design. We, then, demonstrate how this new formulation includes as special cases MOCU-based experimental design methods developed for materials science and genomic networks, when there is experimental error. Most importantly, we show that the classical knowledge gradient and efficient global optimization procedures are specific implementations of MOCU-based experimental design under their modeling assumptions.

An experimental design framework for Markovian gene regulatory networks under stationary control policy

BackgroundA fundamental problem for translational genomics is to find optimal therapies based on gene regulatory intervention. Dynamic intervention involves a control policy that optimally reduces a cost function based on phenotype by externally altering the state of the network over time. When a gene regulatory network (GRN) model is fully known, the problem is addressed using classical dynamic programming based on the Markov chain associated with the network. When the network is uncertain, a Bayesian framework can be applied, where policy optimality is with respect to both the dynamical objective and the uncertainty, as characterized by a prior distribution. In the presence of uncertainty, it is of great practical interest to develop an experimental design strategy and thereby select experiments that optimally reduce a measure of uncertainty.ResultsIn this paper, we employ mean objective cost of uncertainty (MOCU), which quantifies uncertainty based on the degree to which uncertainty degrades the operational objective, that being the cost owing to undesirable phenotypes. We assume that a number of conditional probabilities characterizing regulatory relationships among genes are unknown in the Markovian GRN. In sum, there is a prior distribution which can be updated to a posterior distribution by observing a regulatory trajectory, and an optimal control policy, known as an “intrinsically Bayesian robust” (IBR) policy. To obtain a better IBR policy, we select an experiment that minimizes the MOCU remaining after applying its output to the network. At this point, we can either stop and find the resulting IBR policy or proceed to determine more unknown conditional probabilities via regulatory observation and find the IBR policy from the resulting posterior distribution. For sequential experimental design this entire process is iterated. Owing to the computational complexity of experimental design, which requires computation of many potential IBR policies, we implement an approximate method utilizing mean first passage times (MFPTs) – but only in experimental design, the final policy being an IBR policy.ConclusionsComprehensive performance analysis based on extensive simulations on synthetic and real GRNs demonstrate the efficacy of the proposed method, including the accuracy and computational advantage of the approximate MFPT-based design.

Sequential Experimental Design for Optimal Structural Intervention in Gene Regulatory Networks Based on the Mean Objective Cost of Uncertainty.

Scientists are attempting to use models of ever-increasing complexity, especially in medicine, where gene-based diseases such as cancer require better modeling of cell regulation. Complex models suffer from uncertainty and experiments are needed to reduce this uncertainty. Because experiments can be costly and time-consuming, it is desirable to determine experiments providing the most useful information. If a sequence of experiments is to be performed, experimental design is needed to determine the order. A classical approach is to maximally reduce the overall uncertainty in the model, meaning maximal entropy reduction. A recently proposed method takes into account both model uncertainty and the translational objective, for instance, optimal structural intervention in gene regulatory networks, where the aim is to alter the regulatory logic to maximally reduce the long-run likelihood of being in a cancerous state. The mean objective cost of uncertainty (MOCU) quantifies uncertainty based on the degree to which model uncertainty affects the objective. Experimental design involves choosing the experiment that yields the greatest reduction in MOCU. This article introduces finite-horizon dynamic programming for MOCU-based sequential experimental design and compares it with the greedy approach, which selects one experiment at a time without consideration of the full horizon of experiments. A salient aspect of the article is that it demonstrates the advantage of MOCU-based design over the widely used entropy-based design for both greedy and dynamic programming strategies and investigates the effect of model conditions on the comparative performances.

Intrinsically Bayesian robust Karhunen-Loève compression

Karhunen-Loève (KL) compression is based on the canonical representation of a random process. When compressing to a finite sum, the optimal-MSE m-term summation consists of the KL terms possessing the m largest eigenvalues. This paper considers the situation in which an unknown covariance matrix belongs to an uncertainty class governed by a prior probability distribution. The intrinsically Bayesian robust (IBR) KL compression minimizes the expected MSE over the uncertainty class among all possible m-term KL expansions. We prove that the IBR KL compression is the KL expansion based on the expected covariance matrix over the uncertainty class. We then solve the following experimental design problem: among the unknown covariances, which should be determined to maximally reduce the mean objective cost of uncertainty (MOCU), which in the KL compression setting measures increased MSE resulting from our uncertainty. The IBR KL expansion and optimal experimental design are solved analytically for the Wishart distribution over the uncertainty class, a commonly employed distribution for covariance matrices in Bayesian settings.

Optimal experimental design in the context of canonical expansions

In a wide variety of engineering applications, the mathematical model cannot be fully identified. Therefore, one would like to construct robust operators (filters, classifiers, controllers etc.) that perform optimally relative to incomplete knowledge. Improving model identification through determining unknown parameters can enhance the performance of robust operators. One would like to perform the experiment that provides the most information relative to the engineering objective. The authors present an experimental design framework for parameter estimation in signal processing when the random process model is in the form of canonical expansions. The proposed experimental design is based on the concept of the mean objective cost of uncertainty, which quantifies model uncertainty by taking into account the performance degradation of the designed operator owing to the presence of uncertainty. They provide the general framework for experimental design in the context of canonical expansions and solve it for two major signal processing problems: optimal linear filtering and signal detection.

Optimal experimental design for materials discovery

In this paper, we propose a general experimental design framework for optimally guiding new experiments or simulations in search of new materials with desired properties. The method uses the knowledge of previously completed experiments or simulations to recommend the next experiment which can effectively reduce the pertinent model uncertainty affecting the materials properties. To illustrate the utility of the proposed framework, we focus on a computational problem that utilizes time-dependent Ginzburg-Landau (TDGL) theory for shape memory alloys to calculate the stress-strain profiles for a particular dopant at a given concentration. Our objective is to design materials with the lowest energy dissipation at a specific temperature. The aim of experimental design is to suggest the best dopant and its concentration for the next TDGL simulation. Our experimental design utilizes the mean objective cost of uncertainty (MOCU), which is an objective-based uncertainty quantification scheme that measures uncertainty based upon the increased operational cost it induces. We analyze the performance of the proposed method and compare it with other experimental design approaches, namely random selection and pure exploitation.

Efficient experimental design for uncertainty reduction in gene regulatory networks.

BackgroundAn accurate understanding of interactions among genes plays a major role in developing therapeutic intervention methods. Gene regulatory networks often contain a significant amount of uncertainty. The process of prioritizing biological experiments to reduce the uncertainty of gene regulatory networks is called experimental design. Under such a strategy, the experiments with high priority are suggested to be conducted first.ResultsThe authors have already proposed an optimal experimental design method based upon the objective for modeling gene regulatory networks, such as deriving therapeutic interventions. The experimental design method utilizes the concept of mean objective cost of uncertainty (MOCU). MOCU quantifies the expected increase of cost resulting from uncertainty. The optimal experiment to be conducted first is the one which leads to the minimum expected remaining MOCU subsequent to the experiment. In the process, one must find the optimal intervention for every gene regulatory network compatible with the prior knowledge, which can be prohibitively expensive when the size of the network is large. In this paper, we propose a computationally efficient experimental design method. This method incorporates a network reduction scheme by introducing a novel cost function that takes into account the disruption in the ranking of potential experiments. We then estimate the approximate expected remaining MOCU at a lower computational cost using the reduced networks.ConclusionsSimulation results based on synthetic and real gene regulatory networks show that the proposed approximate method has close performance to that of the optimal method but at lower computational cost. The proposed approximate method also outperforms the random selection policy significantly. A MATLAB software implementing the proposed experimental design method is available at http://gsp.tamu.edu/Publications/supplementary/roozbeh15a/.

Optimal Experimental Design for Gene Regulatory Networks in the Presence of Uncertainty.

Of major interest to translational genomics is the intervention in gene regulatory networks (GRNs) to affect cell behavior; in particular, to alter pathological phenotypes. Owing to the complexity of GRNs, accurate network inference is practically challenging and GRN models often contain considerable amounts of uncertainty. Considering the cost and time required for conducting biological experiments, it is desirable to have a systematic method for prioritizing potential experiments so that an experiment can be chosen to optimally reduce network uncertainty. Moreover, from a translational perspective it is crucial that GRN uncertainty be quantified and reduced in a manner that pertains to the operational cost that it induces, such as the cost of network intervention. In this work, we utilize the concept of mean objective cost of uncertainty (MOCU) to propose a novel framework for optimal experimental design. In the proposed framework, potential experiments are prioritized based on the MOCU expected to remain after conducting the experiment. Based on this prioritization, one can select an optimal experiment with the largest potential to reduce the pertinent uncertainty present in the current network model. We demonstrate the effectiveness of the proposed method via extensive simulations based on synthetic and real gene regulatory networks.

Quantifying the Objective Cost of Uncertainty in Complex Dynamical Systems

Real-world problems often involve complex systems that cannot be perfectly modeled or identified, and many engineering applications aim to design operators that can perform reliably in the presence of such uncertainty. In this paper, we propose a novel Bayesian framework for objective-based uncertainty quantification (UQ), which quantifies the uncertainty in a given system based on the expected increase of the operational cost that it induces. This measure of uncertainty, called MOCU (mean objective cost of uncertainty), provides a practical way of quantifying the effect of various types of system uncertainties on the operation of interest. Furthermore, the proposed UQ framework provides a general mathematical basis for designing robust operators, and it can be applied to diverse applications, including robust filtering, classification, and control. We demonstrate the utility and effectiveness of the proposed framework by applying it to the problem of robust structural intervention of gene regulatory networks, an important application in translational genomics.