High-dimensional Output Research Articles

Computer Model Emulation with High-Dimensional Functional Output in Large-Scale Observing System Uncertainty Experiments

Observing system uncertainty experiments (OSUEs) have been recently proposed as a cost-effective way to perform probabilistic assessment of retrievals for NASA’s Orbiting Carbon Observatory-2 (OCO-2) mission. One important component in the OCO-2 retrieval algorithm is a full-physics forward model that describes the mathematical relationship between atmospheric variables such as carbon dioxide and radiances measured by the remote sensing instrument. This complex forward model is computationally expensive but large-scale OSUEs require evaluation of this model numerous times, which makes it infeasible for comprehensive experiments. To tackle this issue, we develop a statistical emulator to facilitate large-scale OSUEs in the OCO-2 mission. Within each distinct spectral band, the emulator represents radiance output at irregular wavelengths as a linear combination of basis functions and random coefficients. These random coefficients are then modeled with nearest-neighbor Gaussian processes with built-in input dimension reduction via active subspace. The proposed emulator reduces dimensionality in both input space and output space, so that fast computation is achieved within a fully Bayesian inference framework. Validation experiments demonstrate that this emulator outperforms other competing statistical methods and a reduced order model that approximates the full-physics forward model.

Storm hazard analysis over extended geospatial grids utilizing surrogate models

The use of surrogate modeling techniques for storm surge estimation is providing unique opportunities for coastal hazard analysis and risk assessment. Specifically, surrogate models can support a comprehensive estimation of the coastal hazard/risk utilizing ensembles with large number of storm simulations. A critical challenge in this assessment is the high-dimensionality of the output, which needs to be handled efficiently both in terms of computational burden and, especially in the context of automated risk assessment tools, of memory requirements. Though dimensionality reduction techniques, like Principal Component Analysis (PCA), can address this challenge for the surrogate model calibration, the same does not apply for the hazard/risk estimation performed using the trained surrogate model. In this paper, the estimation of coastal risk using large storm ensemble predictions is examined for domains with hundreds of thousands of point locations (grid nodes of the storm surge computational model) for which the corresponding hazard curves need to be provided. This is achieved by exploiting their geospatial distribution and their surge response correlations within the high dimensional original output. K-means clustering is adopted to identify a small subset of nodes to serve as basis for the hazard estimation: instead of calculating the response for the storm ensemble over the entire grid, and then calculating the hazard curves, the hazard curves are first produced for this small subset, and then interpolated over the original grid. Kriging is examined for the latter geospatial interpolation, and its formulation is enhanced to support the desired application, with modifications established for both the calibration stage, as well as the efficient implementation of the interpolation. Additionally, different variants are examined for the clustering stage using information from: (i) the spatial structure of the grid, (ii) the characteristics of the response variability across the geographical domain, and (iii) the combination of the above two. A comprehensive framework is presented integrating both the surrogate model development and the hurricane hazard estimation, with the computational benefits offered in the latter estimation by the use of a small subset of nodes discussed in detail. A validation study is performed utilizing the Coastal Hazards System's (CHS) North Atlantic Coast Comprehensive Study (NACCS) database. It is shown that the achieved numerical efficiency is significant, offering a 50–250-fold reduction of the computational burden with only a moderate impact on the accuracy of the estimated hazard curves.

Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning.

This paper studies the problem of learning the conditional distribution of a high-dimensional output given an input, where the output and input may belong to two different domains, e.g., the output is a photo image and the input is a sketch image. We solve this problem by cooperative training of a fast thinking initializer and slow thinking solver. The initializer generates the output directly by a non-linear transformation of the input as well as a noise vector that accounts for latent variability in the output. The slow thinking solver learns an objective function in the form of a conditional energy function, so that the output can be generated by optimizing the objective function, or more rigorously by sampling from the conditional energy-based model. We propose to learn the two models jointly, where the fast thinking initializer serves to initialize the sampling of the slow thinking solver, and the solver refines the initial output by an iterative algorithm. The solver learns from the difference between the refined output and the observed output, while the initializer learns from how the solver refines its initial output. We demonstrate the effectiveness of the proposed method on various conditional learning tasks, e.g., class-to-image generation, image-to-image translation, and image recovery. The advantage of our method over GAN-based methods is that our method is equipped with a slow thinking process that refines the solution guided by a learned objective function.

Deep coregionalization for the emulation of simulation-based spatial-temporal fields

Data-driven surrogate models are widely used for applications such as design optimization and uncertainty quantification, where repeated evaluations of an expensive simulator are required. For most partial differential equation (PDE) simulations, the outputs of interest are often spatial or spatial-temporal fields, leading to very high-dimensional outputs. Despite the success of existing data-driven surrogates for high-dimensional outputs, most methods require a significant number of samples to cover the response surface in order to achieve a reasonable degree of accuracy. This demand makes the idea of surrogate models less attractive considering the high-computational cost to generate the data. To address this issue, we exploit the multifidelity nature of a PDE simulation and introduce deep coregionalization, a Bayesian nonparametric autoregressive framework for efficient emulation of spatial-temporal fields. To effectively extract the output correlations in the context of multifidelity data, we develop a novel dimension reduction technique, residual principal component analysis. Our model can simultaneously capture the rich output correlations and the fidelity correlations and make high-fidelity predictions with only a small number of expensive, high-fidelity simulation samples. We show the advantages of our model in three canonical PDE models and a fluid dynamics problem. The results show that the proposed method can not only approximate simulation results with significantly less cost (by bout 10%-25%) but also further improve model accuracy.

A new multi-task learning framework for fuel cell model outputs in high-dimensional spaces

In this paper we develop a multi-output, multi-task framework for approximating the outputs of complex physics-based computer models of fuel cells and batteries. For applications that require real-time or a high number of runs, such as optimization and control, the original computer model will be impractical, making such approximations necessary. It is normally the case that there are several quantities of interest (different tasks) from the model, such as the potential, reactant distributions and the cell voltage charge–discharge profile. We overcome several challenges in this paper: dealing with very high dimensional outputs (e.g., a distribution defined at many grid points), accounting for correlations between tasks, and accounting for noise in the outputs or missing information. We combine a multivariate Gaussian process (GP) model based on dimension reduction with a linear model of coregionalisation (LMC) to account for the between-task covariances. The LMC applies to coefficients in a low-dimensional subspace (each coefficient corresponding to a particular task), which results in highly-efficient training. We test the framework on two different 3-d hydrogen fuel cell models and compare the results to several alternative multivariate GP approaches. The results reveal that our framework is more accurate and also more computationally efficient.

Statistical retrieval of atmospheric profiles with deep convolutional neural networks

Infrared atmospheric sounders, such as IASI, provide an unprecedented source of information for atmosphere monitoring and weather forecasting. Sensors provide rich spectral information that allows retrieval of temperature and moisture profiles. From a statistical point of view, the challenge is immense: on the one hand, “underdetermination” is common place as regression needs to work on high dimensional input and output spaces; on the other hand, redundancy is present in all dimensions (spatial, spectral and temporal). On top of this, several noise sources are encountered in the data.In this paper, we present for the first time the use of convolutional neural networks for the retrieval of atmospheric profiles from IASI sounding data. The first step of the proposed pipeline performs spectral dimensionality reduction taking into account the signal to noise characteristics. The second step encodes spatial and spectral information, and finally prediction of multidimensional profiles is done with deep convolutional networks. We give empirical evidence of the performance in a wide range of situations. Networks were trained on orbits of IASI radiances and tested out of sample with great accuracy over competing approximations, such as linear regression (+32%). We also observed an improvement in accuracy when predicting over clouds, thus increasing the yield by 34% over linear regression. The proposed scheme allows us to predict related variables from an already trained model, performing transfer learning in a very easy manner. We conclude that deep learning is an appropriate learning paradigm for statistical retrieval of atmospheric profiles.

Efficient dimension reduction and surrogate-based sensitivity analysis for expensive models with high-dimensional outputs

Sensitivity analysis has been widely used to gain more insights on complex system behavior, to facilitate model reduction, system design and decision making. Typically, sensitivity analysis entails many evaluations of the system model. For expensive system models with high-dimensional outputs, direct adoption of such models for sensitivity analysis poses significant challenges in computational effort and memory requirements. To address these challenges, this paper proposes an efficient sensitivity analysis approach. The proposed method uses surrogate model to replace the expensive model for sensitivity analysis, and tackle the problem of building surrogate models for high-dimensional outputs through surrogate model integrated with dimension reduction. More specifically, the proposed method first uses surrogate models in low-dimensional latent output space to efficiently calculate the relevant covariance matrices for the low-dimensional latent outputs, and then directly establishes the sensitivity indices for the original high-dimensional output based on these covariance matrices and the derived transformation. Two examples are presented to demonstrate the efficiency and accuracy of the proposed method.

Information-theoretic analysis of multivariate single-cell signaling responses.

Mathematical methods of information theory appear to provide a useful language to describe how stimuli are encoded in activities of signaling effectors. Exploring the information-theoretic perspective, however, remains conceptually, experimentally and computationally challenging. Specifically, existing computational tools enable efficient analysis of relatively simple systems, usually with one input and output only. Moreover, their robust and readily applicable implementations are missing. Here, we propose a novel algorithm, SLEMI—statistical learning based estimation of mutual information, to analyze signaling systems with high-dimensional outputs and a large number of input values. Our approach is efficient in terms of computational time as well as sample size needed for accurate estimation. Analysis of the NF-κB single—cell signaling responses to TNF-α reveals that NF-κB signaling dynamics improves discrimination of high concentrations of TNF-α with a relatively modest impact on discrimination of low concentrations. Provided R-package allows the approach to be used by computational biologists with only elementary knowledge of information theory.

High-throughput multicolor optogenetics in microwell plates.

Optogenetic probes can be powerful tools for dissecting complexity in cell biology, but there is a lack of instrumentation to exploit their potential for automated, high-information-content experiments. This protocol describes the construction and use of the optoPlate-96, a platform for high-throughput three-color optogenetics experiments that allows simultaneous manipulation of common red- and blue-light-sensitive optogenetic probes. The optoPlate-96 enables illumination of individual wells in 96-well microwell plates or in groups of wells in 384-well plates. Its design ensures that there will be no cross-illumination between microwells in 96-well plates, and an active cooling system minimizes sample heating during light-intensive experiments. This protocol details the steps to assemble, test, and use the optoPlate-96. The device can be fully assembled without specialized equipment beyond a 3D printer and a laser cutter, starting from open-source design files and commercially available components. We then describe how to perform a typical optogenetics experiment using the optoPlate-96 to stimulate adherent mammalian cells. Although optoPlate-96 experiments are compatible with any plate-based readout, we describe analysis using quantitative single-cell immunofluorescence. This workflow thus allows complex optogenetics experiments (independent control of stimulation colors, intensity, dynamics, and time points) with high-dimensional outputs at single-cell resolution. Starting from 3D-printed and laser-cut components, assembly and testing of the optoPlate-96 can be accomplished in 3-4 h, at a cost of ~$600. A full optoPlate-96 experiment with immunofluorescence analysis can be performed within ~24 h, but this estimate is variable depending on the cell type and experimental parameters.

Compression of recurrent neural networks for efficient language modeling

Recurrent neural networks have proved to be an effective method for statistical language modeling. However, in practice their memory and run-time complexity are usually too large to be implemented in real-time offline mobile applications. In this paper we consider several compression techniques for recurrent neural networks including Long–Short Term Memory models. We make particular attention to the high-dimensional output problem caused by the very large vocabulary size. We focus on effective compression methods in the context of their exploitation on devices: pruning, quantization, and matrix decomposition approaches (low-rank factorization and tensor train decomposition, in particular). For each model we investigate the trade-off between its size, suitability for fast inference and perplexity. We propose a general pipeline for applying the most suitable methods to compress recurrent neural networks for language modeling. It has been shown in the experimental study with the Penn Treebank (PTB) dataset that the most efficient results in terms of speed and compression–perplexity balance are obtained by matrix decomposition techniques.

Patient-Specific Seizure Detection Method using Hybrid Classifier with Optimized Electrodes.

In this paper the EEG signal is analyzed by reconstructing the time series EEG signal in High dimensional Phase Space. The computational complexity in higher dimension is reduced by Principal Component Analysis for the High dimensional Phase Space output. Poincare sectioning is done for the first and second Principal Components (PCs). The intersection points of PCs and the Poincare section are collected and used for features calculation. Two layer of classification is done using SVM as first layer and Naive Bayes as second layer. The proposed methodology is evaluated using the CHB-MIT database for 23 subjects. The results are obtained using different channel combinations of EEG signal and highest of 95.63% accuracy, 95.7% sensitivity and 96.55% specificity is obtained for 12 electrode combinations which include electrodes from parietal and occipital lobes. This infers that most of the subjects have dysfunction in hearing (controlled by parietal) and vision (controlled by occipital) during the time of seizure. This GUI has channel selection option and seizure detection for every channel (23) for every 1s.

Investigation of impact of shoreline alteration on coastal hydrodynamics using Dimension REduced Surrogate based Sensitivity Analysis

To inform the decision-making of coastal protection against sea level rise (SLR), we have to estimate the impact of shoreline alterations on the hydrodynamics. This task involves estimating of a large number of shoreline decision combinations. Here we present a sensitivity analysis based approach to understand how the variation in the shoreline, for example, due to construction of seawalls at different location along the shoreline, would impact the water levels over the coastal region under SLR. To facilitate efficient sensitivity analysis for expensive high-fidelity numerical models with high-dimensional outputs, we propose a Dimension REduced Surrogate based Sensitivity Analysis (DRESSA) method. DRESSA uses Principal Component Analysis (PCA) to exploit the correlation in the high-dimensional outputs to find a low-dimensional latent output representation, then builds a surrogate model for the latent outputs based on a small number of runs of the high-fidelity numerical model. In the end, DRESSA first establishes relevant covariance matrices for the low-dimensional latent outputs using the surrogate model, and then directly establishes sensitivity indexes for the high-dimensional outputs using these covariance matrices and the derived transformation between sensitivity in latent space and original space. We applied this method to generate sensitivity maps and investigate the impact of different containment strategies on peak water level (PWL) over the entire San Francisco Bay under SLR.

Conversion of Mersenne Twister to double-precision floating-point numbers

The 32-bit Mersenne Twister generator MT19937 is a widely used random number generator. To generate numbers with more than 32 bits in bit length, and particularly when converting into 53-bit double-precision floating-point numbers in [0,1) in the IEEE 754 format, the typical implementation concatenates two successive 32-bit integers and divides them by a power of 2. In this case, the 32-bit MT19937 is optimized in terms of its equidistribution properties (the so-called dimension of equidistribution with v-bit accuracy) under the assumption that one will mainly be using 32-bit output values, and hence the concatenation sometimes degrades the dimension of equidistribution compared with the simple use of 32-bit outputs. In this paper, we analyze such phenomena by investigating hidden F2-linear relations among the bits of high-dimensional outputs. Accordingly, we report that MT19937 with a specific lag set fails several statistical tests, such as the overlapping collision test, matrix rank test, and Hamming independence test.

Projection-based model reduction: Formulations for physics-based machine learning

This paper considers the creation of parametric surrogate models for applications in science and engineering where the goal is to predict high-dimensional output quantities of interest, such as pressure, temperature and strain fields. The proposed methodology develops a low-dimensional parametrization of these quantities of interest using the proper orthogonal decomposition (POD), and combines this parametrization with machine learning methods to learn the map between the input parameters and the POD expansion coefficients. The use of particular solutions in the POD expansion provides a way to embed physical constraints, such as boundary conditions and other features of the solution that must be preserved. The relative costs and effectiveness of four different machine learning techniques—neural networks, multivariate polynomial regression, k-nearest-neighbors and decision trees—are explored through two engineering examples. The first example considers prediction of the pressure field around an airfoil, while the second considers prediction of the strain field over a damaged composite panel. The case studies demonstrate the importance of embedding physical constraints within learned models, and also highlight the important point that the amount of model training data available in an engineering setting is often much less than it is in other machine learning applications, making it essential to incorporate knowledge from physical models.

Multifidelity coKriging for High-Dimensional Output Functions with Application to Hypersonic Airloads Computation

An accurate surrogate model is capable of reproducing high-fidelity results at a fraction of the computational cost. coKriging is an efficient method for constructing surrogate models when two levels of fidelity (low and high) tools are available. This paper extends the functionality of coKriging. First, a practical implementation to high-dimensional output functions is developed by introducing POD-coKriging. Second, coKriging is reformulated to allow different sizes of input/output variables between low- and high-fidelity tools. The extended POD-coKriging method is validated against an analytical problem, and its performance for high-dimensional output functions is demonstrated with the prediction of surface pressure distribution on a two-dimensional deflected panel in hypersonic flow. Finally, properties of the extended POD-coKriging were used to estimate the total cost reduction of sample generation, and it is shown that an order-of-magnitude cost reduction is achievable for practical cases.

Uncertainty Quantification for Flow and Transport in Highly Heterogeneous Porous Media Based on Simultaneous Stochastic Model Dimensionality Reduction

Groundwater flow models are usually subject to uncertainty as a consequence of the random representation of the conductivity field. In this paper, we use a Gaussian process model based on the simultaneous dimension reduction in the conductivity input and flow field output spaces in order quantify the uncertainty in a model describing the flow of an incompressible liquid in a random heterogeneous porous medium. We show how to significantly reduce the dimensionality of the high-dimensional input and output spaces while retaining the qualitative features of the original model, and secondly how to build a surrogate model for solving the reduced-order stochastic model. A Monte Carlo uncertainty analysis on the full-order model is used for validation of the surrogate model.

Probabilistic sensitivity analysis for multivariate model outputs with applications to Li-ion batteries

Full battery models are highly complex, which limits their application to tasks such as optimization and uncertainty quantification. To lower the computational burden, sensitivity analysis (SA) can be used as a precursor to identify the most important parameters in the model, but SA itself relies on a high number of full model evaluations, which has motivated the use of emulators. For high-dimensional output problems, emulators are challenging to construct. In this paper we develop a probabilistic framework for SA of high-dimensional output models using a Gaussian process emulator based on dimensionality reduction. This allows us to perform SA under uncertainty for multi-ouput problems, providing error bounds for the emulator predictions of sensitivity measures. We show how this can be achieved using Monte Carlo sampling or possibly by using semi-analytical expressions with highly efficient sampling. Moreover, we can perform SA for multivariate outputs by ranking the sensitivity measures related to (uncorrelated) coefficients in a basis for the output space.

Exposing System and Model Disparity and Agreement Using Wavelets

Model verification and validation (V&V) remain a critical step in the simulation model development process. A model requires verification to ensure that it has been correctly transitioned from a conceptual form to a computerized form. A model also requires validation to substantiate the accurate representation of the system it is meant to simulate. Validation assessments are complex when the system and model both generate high-dimensional functional output. To handle this complexity, this paper reviews several wavelet-based approaches for assessing models of this type and introduces a new concept for highlighting the areas of contrast and congruity between system and model data. This concept identifies individual wavelet coefficients that correspond to the areas of discrepancy between the system and model.

Meta-modeling of ADMS-Urban by dimension reduction and emulation

ADMS-Urban is a non-linear, static, urban air quality model, with high-dimensional outputs. A simulation of NO2 and PM10 concentrations every hour during a full year and over an entire city can take dozens of days of computations, which greatly limits the range of methods that can be applied to the model, especially for uncertainty quantification. This work presents a method to replace the complete model, ADMS-Urban, with a meta-model or surrogate model, i.e., a reasonably close approximation of ADMS-Urban whose computational cost is negligible. When the emissions are formulated as a function of the day and the hour, the complete-model inputs essentially contain a few scalar values, to describe the meteorological conditions, the background pollution and the target date. The complete-model outputs are first projected onto a reduced subspace. The relations between the projection coefficients and the low-dimensional inputs are then emulated by a fast statistical emulator, based on Kriging or radial basis functions (RBF). The mean error between the meta-model and ADMS-Urban is 22% with Kriging and 27% with RBF for NO2, and 14% with Kriging and 20% with RBF for PM10. The meta-model performs as well as ADMS-Urban when compared to the observations. Its computational cost is almost negligible to compute the concentrations at a given hour and date for an entire city: 50 ms with RBF and 150 ms with Kriging to simulate 1 h on one core, while the complete model requires 8 min on 16 cores.

Surrogate modelling for the prediction of spatial fields based on simultaneous dimensionality reduction of high-dimensional input/output spaces.

Time-consuming numerical simulators for solving groundwater flow and dissolution models of physico-chemical processes in deep aquifers normally require some of the model inputs to be defined in high-dimensional spaces in order to return realistic results. Sometimes, the outputs of interest are spatial fields leading to high-dimensional output spaces. Although Gaussian process emulation has been satisfactorily used for computing faithful and inexpensive approximations of complex simulators, these have been mostly applied to problems defined in low-dimensional input spaces. In this paper, we propose a method for simultaneously reducing the dimensionality of very high-dimensional input and output spaces in Gaussian process emulators for stochastic partial differential equation models while retaining the qualitative features of the original models. This allows us to build a surrogate model for the prediction of spatial fields in such time-consuming simulators. We apply the methodology to a model of convection and dissolution processes occurring during carbon capture and storage.