High-dimensional Output Research Articles

Single Cell Sequencing and Spatial Transcriptomics in Kidney Transplantation

Abstract Purpose of Review Progress in biology and medicine relies on technological advances. The ability to examine gene expression from single cells in tissues with complex architecture is now possible with the advent of single cell sequencing and spatial transcriptomics methods. This review will outline how these methods have been applied to kidney transplantation and their inherent challenges and limitations. Recent Findings The transcriptomes of single cells can now be defined in health and disease allowing for the differences to be investigated. Advanced in situ sequencing and capture methodologies now add spatial resolution to single cell transcriptomics. However, such powerful methods bring challenges, most notably the associated computational burden that is required to make the resulting high-dimensional output data interpretable. A second challenge is that of cost. Thankfully, as with many new technologies’ costs will drop over time making these methods available to more research groups. Summary The sequencing and analysis of gene expression at the single cell level with spatial information retained is now feasible. These methods are major advances for biomedical research and bring huge potential with regards to the better understanding of disease at cellular and organ levels.

Active learning for adaptive surrogate model improvement in high-dimensional problems

This paper investigates a novel approach to efficiently construct and improve surrogate models in problems with high-dimensional input and output. In this approach, the principal components and corresponding features of the high-dimensional output are first identified. For each feature, the active subspace technique is used to identify a corresponding low-dimensional subspace of the input domain; then a surrogate model is built for each feature in its corresponding active subspace. A low-dimensional adaptive learning strategy is proposed to identify training samples to improve the surrogate model. In contrast to existing adaptive learning methods that focus on a scalar output or a small number of outputs, this paper addresses adaptive learning with high-dimensional input and output, with a novel learning function that balances exploration and exploitation, i.e., considering unexplored regions and high-error regions, respectively. The adaptive learning is in terms of the active variables in the low-dimensional space, and the newly added training samples can be easily mapped back to the original space for running the expensive physics model. The proposed method is demonstrated for the numerical simulation of an additive manufacturing part, with a high-dimensional field output quantity of interest (residual stress) in the component that has spatial variability due to the stochastic nature of multiple input variables (including process variables and material properties). Various factors in the adaptive learning process are investigated, including the number of training samples, range and distribution of the adaptive training samples, contributions of various errors, and the importance of exploration versus exploitation in the learning function.

Surrogate-based modeling and sensitivity analysis of future European electricity spot market prices

Europe is transitioning towards a sustainable energy economy. Planning these future power and energy systems is multifaceted and complex, requiring long-term projections of future electricity market dynamics. The prognostication of future prices is crucial for power system planning, as these prices will drive investments and impact incentives to build and improve new systems. The price of electricity in different regions impacts transmission investment. Energy systems models have high-dimensional output and are expensive to evaluate, leading to unique challenges in managing the uncertainty in model assumptions. This study presents a novel methodology for determining which cost assumptions are most impactful on the output of a detailed model of the European energy system. Because this model is computationally intensive, only a small number of samples are available to us. A multiple-output support vector regression is used as a surrogate of predicted electricity prices timeseries. We find that solar and wind costs are the primary drivers of electricity prices, but natural gas costs generally drive peak electricity prices and those far in the future.

A dimension reduction-based Kriging modeling method for high-dimensional time-variant uncertainty propagation and global sensitivity analysis

Surrogate models have been widely used in the uncertainty propagation and global sensitivity analysis of complex evaluation-expensive engineering problems. However, the construction of high-accuracy surrogate models for time-variant problems with high-dimensional inputs and outputs using a small number of training samples remains a challenge. To address this challenge, we propose a dimension reduction-based Kriging modeling (KMDR) method for high-dimensional time-variant uncertainty propagation and global sensitivity analysis. Singular value decomposition is performed on the original time-variant response to extract principal components from high-dimensional outputs. And the improved sufficient dimension reduction (ISDR) is performed on high-dimensional inputs to identify the latent input space with respect to each principal component of outputs. A ladle estimator with a rigorous mathematical definition is then employed to determine the number of principal components of the outputs and the dimensionalities of the latent input spaces. The ladle estimator considers variabilities in both eigenvalues and eigenvectors of a matrix and can determine the latent dimensionality more accurately and efficiently than existing approaches. Subsequently, Kriging models between high-dimensional inputs and each principal component of the outputs are constructed based on a newly devised Kriging kernel, which embeds the information of the ISDR into the kernel function and can achieve higher accuracy than directly constructing Kriging models between latent inputs and outputs. In addition, a generalized variance-based sensitivity index, which can quantify the effects of the inputs on the overall time-variant response, is defined and computed directly from Sobol’ sensitivity indices of the inputs at each time node. Finally, the surrogate model of the time-variant system constructed by the KMDR is directly adopted for efficient time-variant uncertainty propagation and global sensitivity analysis. Several examples demonstrate that the proposed approach can construct more accurate surrogate models and obtain more accurate time-variant uncertainty propagation and global sensitivity analysis results than existing methods with a small training set.

Memory-based Distribution Shift Detection for Learning Enabled Cyber-Physical Systems with Statistical Guarantees

Incorporating learning based components in the current state-of-the-art cyber-physical systems (CPS) has been a challenge due to the brittleness of the underlying deep neural networks. On the bright side, if executed correctly with safety guarantees, this has the ability to revolutionize domains like autonomous systems, medicine, and other safety-critical domains. This is because it would allow system designers to use high-dimensional outputs from sensors like camera and LiDAR. The trepidation in deploying systems with vision and LiDAR components comes from incidents of catastrophic failures in the real world. Recent reports of self-driving cars running into difficult to handle scenarios is ingrained in the software components which handle such sensor inputs. The ability to handle such high-dimensional signals is due to the explosion of algorithms which use deep neural networks. Sadly, the reason behind the safety issues is also due to deep neural networks themselves. The pitfalls occur due to possible over-fitting and lack of awareness about the blind spots induced by the training distribution. Ideally, system designers would wish to cover as many scenarios during training as possible. However, achieving a meaningful coverage is impossible. This naturally leads to the following question: is it feasible to flag out-of-distribution (OOD) samples without causing too many false alarms? Such an OOD detector should be executable in a fashion that is computationally efficient. This is because OOD detectors often are executed as frequently as the sensors are sampled. Our aim in this article is to build an effective anomaly detector. To this end, we propose the idea of a memory bank to cache data samples which are representative enough to cover most of the in-distribution data. The similarity with respect to such samples can be a measure of familiarity of the test input. This is made possible by an appropriate choice of distance function tailored to the type of sensor we are interested in. Additionally, we adapt conformal anomaly detection framework to capture the distribution shifts with a guarantee of false alarm rate. We report the performance of our technique on two challenging scenarios: a self-driving car setting implemented inside the simulator CARLA with image inputs and autonomous racing car navigation setting with LiDAR inputs. From the experiments, it is clear that a deviation from the in-distribution setting can potentially lead to unsafe behavior. It should be noted that not all OOD inputs lead to precarious situations in practice, but staying in-distribution is akin to staying within a safety bubble and predictable behavior. An added benefit of our memory-based approach is that the OOD detector produces interpretable feedback for a human designer. This is of utmost importance since it recommends a potential fix for the situation as well. In other competing approaches, such feedback is difficult to obtain due to reliance on techniques which use variational autoencoders.

A compressed 2D-DOA and polarization estimation algorithm for mmWave polarized massive MIMO systems

Due to providing greater degrees-of-freedom (DOFs) and securer communication guarantees than traditional scalar array, polarized massive multiple-input multiple-output (MIMO) technique offers a prospective insight into millimeter-wave (mmWave) communication. In this paper, a compressed two-dimensional (2D) direction-of-arrival (DOA) and polarization estimation algorithm combining the compressive sampling propagator method with reduced-dimension multiple signal classification (CSPM-RDMUSIC) is developed for mmWave polarized massive MIMO systems. First, a synchronous compressive network is constructed to compress the high-dimensional output into a low-dimensional one. Thereafter, a propagator is structured to acquire the signal subspace. Subsequently, accurate 2D-DOA and polarization estimation are achieved using a coarse-refined strategy, where the vector cross-product technology is investigated for coarse 2D-DOA estimation and then RDMUSIC is exploited to perform 2D local search for accurate parameter estimation. Owing to integrating the CSPM framework with the RDMUSIC strategy, the developed approach provides high-precision parameter estimation with relative computational economy, supporting mmWave communication demands. What's more, it's suitable for arbitrary array geometry with high flexibility. Several experimental results validate the superiorities of the developed framework.

Motion Learning and Generalization of Musculoskeletal Robot Using Gain Primitives

Organisms have an innate ability to rapidly produce diverse and flexible movement. Biological motor systems are composed of highly redundant muscle actuators and have strongly nonlinear dynamic characteristics. How organisms cope with such complex system is still an open question. Motor primitive theory proposed by neuroscientists has provided a convincing explanation for this extraordinary ability and have supported by several biological evidence. In this paper, based on the gain primitive model of cortical network proposed by neuroscientists, new algorithms for learning and combining gain primitives were designed to control musculoskeletal and robotic system with highly redundant actuators. It can realize dimensionality reduction of controll from the number of actuators to the number of primitives. A parameter adaptation algorithm inspired by monoamines modulation mechanism was applied to improve the learning efficiency of gain primitive. Learned motion experiences and primitives were introduced to effectively generate new movements. Validation experiments were carried out on a musculoskeletal model with 9 muscles and an articulated robot Baxter with redundant actuators. The experimental results demonstrated that the recurrent neural network modulated by gain primitives can generate high-dimensional control signals and realize efficient motion generalization for practical systems. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper applied a biologically plausible control method with gain primitive to practical systems with highly redundant actuators. To reduce the cumulative error occurring in learning gain primitive for control networks with high dimensional output, a bioinspired Hebbian learning rule was applied to train a recurrent neural network for optimizing the gain primitive. During the training process, systematic entropy was introduced as an index to adaptively adjust learning parameters, so as to reduce the exploration cost and improve the learning stability of the system. To realize easy implementation and generalization, prior motion experiences were introduced into the initialization of the optimization algorithm, and motion generalization was realized by using the particle swarm optimization algorithm. The algorithms designed in this paper are applicable to complex systems with highly redundant actuators such as humanoid robots.

Optical Extreme Learning Machines with Atomic Vapors

Extreme learning machines explore nonlinear random projections to perform computing tasks on high-dimensional output spaces. Since training only occurs at the output layer, the approach has the potential to speed up the training process and the capacity to turn any physical system into a computing platform. Yet, requiring strong nonlinear dynamics, optical solutions operating at fast processing rates and low power can be hard to achieve with conventional nonlinear optical materials. In this context, this manuscript explores the possibility of using atomic gases in near-resonant conditions to implement an optical extreme learning machine leveraging their enhanced nonlinear optical properties. Our results suggest that these systems have the potential not only to work as an optical extreme learning machine but also to perform these computations at the few-photon level, paving opportunities for energy-efficient computing solutions.

Industrial Data Modeling With Low-Dimensional Inputs and High-Dimensional Outputs

Data-driven modeling will be complicated for a process if the output quality indices are defined in a high-dimensional space, e.g., a quality distribution. In this work, a novel probabilistic modeling method is proposed for industrial processes with low-dimensional inputs and high-dimensional outputs. First, based on a limited sample set, the variational autoencoder (VAE) is applied to extract features of the high-dimensional outputs. Next, a Gaussian Process (GP) model is established on the sub-manifold space defined by the low-dimensional features, and the high-dimensional predictions can be obtained through the VAE reverse procedure. Finally, a deep composing kernel strategy is developed to capture the nonlinearity and correlation hidden in the features. It can significantly improve the generalization performance of the GP model. The effectiveness of the proposed modeling algorithm is demonstrated by applications in a continuous crystallizer system and an ethylene homo-polymerization system.

Binning high-dimensional classifier output for HEP analyses through a clustering algorithm

Binning high-dimensional classifier output for HEP analyses through a clustering algorithm

Scalable Bayesian optimization with randomized prior networks

Several fundamental problems in science and engineering consist of global optimization tasks involving unknown high-dimensional (black-box) functions that map a set of controllable variables to the outcomes of an expensive experiment. Bayesian Optimization (BO) techniques are known to be effective in tackling global optimization problems using a relatively small number objective function evaluations, but their performance suffers when dealing with high-dimensional outputs. To overcome the major challenge of dimensionality, here we propose a deep learning framework for BO and sequential decision making based on bootstrapped ensembles of neural architectures with randomized priors. Using appropriate architecture choices, we show that the proposed framework can approximate functional relationships between design variables and quantities of interest, even in cases where the latter take values in high-dimensional vector spaces or even infinite-dimensional function spaces. In the context of BO, we augmented the proposed probabilistic surrogates with re-parameterized Monte Carlo approximations of multiple-point (parallel) acquisition functions, as well as methodological extensions for accommodating black-box constraints and multi-fidelity information sources. We test the proposed framework against state-of-the-art methods for BO and demonstrate superior performance across several challenging tasks with high-dimensional outputs, including a constrained multi-fidelity optimization task involving shape optimization of rotor blades in turbo-machinery.

Research on surrogate model of dam numerical simulation with multiple outputs based on adaptive sampling.

Dam numerical simulation is an important method to research the dam structural behavior, but it often takes a lot of time for calculation when facing problems that require many simulations, such as structural parameter back analysis. The surrogate model is widely used as a technology to reduce computational cost. Although various methods have been widely investigated, there are still problems in designing the surrogate model's optimal Design of Experiments (DoE). In addition, most of the current DoE focuses on establishing a single-output problem. Designing a reasonable DoE for high-dimensional outputs is also a problem that needs to be solved. Based on the above issues, this research proposes a sequential surrogate model based on the radial basis function model (RBFM) with multi-outputs adaptive sampling. The benchmark function demonstrates the applicability of the proposed method to single-input & multi-outputs and multi-inputs & multi-outputs problems. Then, this method is applied to establishing a surrogate model for dam numerical simulation with multi-outputs. The result demonstrates that the proposed technique can be sampled adaptively and samples can be targeted based on the function form of the surrogate model, which significantly reduces the required sampling and calculation cost.

An Efficient Deep Learning-Based Workflow for CO2 Plume Imaging With Distributed Pressure and Temperature Measurements

Summary Geologic carbon dioxide (CO2) sequestration has received significant attention from the scientific community as a response to global warming due to greenhouse gas emissions. Effective monitoring of CO2 plume is critical to CO2 storage safety throughout the life cycle of a geologic CO2 sequestration project. Although simulation-based techniques such as history matching can be used for predicting the evolution of underground CO2 saturation, the computational cost of high-fidelity simulations can be prohibitive. Recent development in data-driven models can provide a viable alternative for rapid CO2 plume imaging. Here, we present a novel deep learning–based workflow that can efficiently visualize CO2 plume in near real time. Our deep learning framework utilizes field measurements, such as downhole pressure, distributed pressure, and temperature, as input to visualize the subsurface CO2 plume images. However, the high output dimension of CO2 plume images makes the training inefficient. We address this challenge in two ways: First, we output a single CO2 onset time map rather than multiple saturation maps at different times; second, we apply an autoencoder-decoder network to identify lower-dimensional latent variables that compress high-dimensional output images. The “onset time” is the calendar time when the CO2 saturation at a given location exceeds a specified threshold value. In our approach, a deep learning–based regression model is trained to predict latent variables of the autoencoder-decoder network. Subsequently, the latent variables are used as inputs of the trained decoder network to generate the 3D onset time image, visualizing the evolving CO2 plume in near real time. The power and efficacy of our approach are demonstrated using both synthetic and field-scale applications. We first validate the deep learning–based CO2 plume imaging workflow using a 2D synthetic example. Next, the visualization workflow is applied to a 3D field-scale reservoir to demonstrate the robustness and efficiency of the workflow. The monitoring data set consists of distributed temperature sensing (DTS) data acquired at a monitoring well, flowing bottomhole pressure (BHP) data at the injection well, and time-lapse pressure measurements at several locations along the monitoring well. Our approach is also extended to efficiently evaluate the uncertainty of predicted CO2 plume images. Additionally, an efficient workflow for optimizing data acquisition and measurement type is demonstrated using our deep learning–based framework. The novelty of this work is the development and application of a unique and efficient deep learning–based subsurface visualization workflow for the spatial and temporal migration of the CO2 plume. The efficiency and flexibility of the data-driven workflow make our approach suitable for field-scale applications.

A comparative analysis of dimensionality reduction surrogate modeling techniques for full human body finite element impact simulations

Fast-running surrogate computational models (simpler computational models) have been successfully used to replace time-intensive finite element models. However, it is unclear how well they perform in accurately and efficiently replicating complex, full human body finite element models. Here we survey several surrogate modeling techniques and assess their accuracy in predicting full strain fields of tissues of interest during a highly dynamic behind armor blunt trauma impact to the liver. We found that coupling dimensionality reduction on the high-dimensional output space (principal component analysis or autoencoders) with machine learning techniques (Gaussian Process Regression or multi-output neural networks) provides a framework capable of accurately and efficiently replacing complex full human body models. It was found that these surrogate models can successfully predict the strain fields (<10% average strain error) of select tissues during a nonlinear impact event but careful consideration should be given to element parsing and modeling technique.

Robust optimization for functional multiresponse in 3D printing process

Computer models are commonly used to simulate the functional relationships between inputs and outputs for quality design in 3D printing. However, the high-dimensional outputs of functional multiresponse make it challenging to develop the simulation model and perform robust optimization. This paper proposes a novel optimization method with an additive multiresponse Gaussian process model for dealing with functional multiresponse optimization problems. First, an additive covariance function is constructed to capture the correlation of the temporal inputs. Second, the Markov Chain Monte Carlo sampling technique is adopted to determine the simulation model and quantify the uncertainty. Finally, the optimization model is constructed by integrating the quality loss function and interval analysis method, and the Bayesian optimization algorithm is used to obtain the optimal solution. A numerical simulation example and a 3D printing case study are used to illustrate the effectiveness of the proposed method. The comparison results show that the responses of the proposed method are closer to the targets than the current ones, and all fall within the specified interval.

Improved metamodels for predicting high-dimensional outputs by accounting for the dependence structure of the latent variables: application to marine flooding

Improved metamodels for predicting high-dimensional outputs by accounting for the dependence structure of the latent variables: application to marine flooding

Efficient global sensitivity analysis for high-dimensional outputs combining data-driven probability models and dimensionality reduction

• GSA for applications involving complex numerical models and high-dimensional outputs is examined • Two different, recently developed, GSA formulations are combined • Principal component analysis (PCA) is firstly adopted as a dimensionality reduction technique. • Relevant latent output GSA statistics are then calculated by extending probability model-based GSA. • Extension pertains to estimation of necessary covariance statistics This paper examines the efficient variance-based global sensitivity analysis (GSA), quantified by estimating first-/higher-order and total-effect Sobol’ indices, for applications involving complex numerical models and high-dimensional outputs. Two different, recently developed, techniques are combined to address the associated challenges. Principal component analysis (PCA) is first considered as a dimensionality reduction technique. The GSA for the original output vector is then formulated by calculating variance and covariance statistics for the low-dimensional latent output space identified by PCA. These statistics are efficiently approximated by extending recent work on data-driven, probability model-based GSA (PM-GSA). The extension, constituting the main novel contribution of this work, pertains to the estimation of covariance statistics beyond the variance statistics examined in the original PM-GSA formulation. Specifically, a Gaussian mixture model (GMM) is developed to approximate the joint probability density function between some subset of the input vector, and each latent output, or each pair of latent outputs. The GMM is then utilized to estimate the aforementioned statistics. Results across two natural hazards engineering examples show that the dimensionality reduction and transformation of output space established through PCA do not impact the overall accuracy of the PM-GSA, and that the proposed implementation accommodates highly-efficient GSA estimates.

An efficient method for time-dependent reliability problems with high-dimensional outputs based on adaptive dimension reduction strategy and surrogate model

Time-dependent reliability analysis with high dimensional outputs is a challenge because of ‘curse of dimensionality’ and accurate reliability estimation over entire time interval is computationally expensive. In this paper, an efficient time-dependent reliability analysis method is proposed for systems with high dimensional outputs based on adaptive dimension reduction strategy and Kriging. The adaptive Kriging model is constructed in low-dimensional space after performing principal component analysis (PCA) on original time-dependent output. A new learning function and corresponding stopping criterion are developed as the guideline for selecting training samples at each iteration. The proposed learning function focuses on prediction accuracy over the entire time interval and the stopping criterion is directly linked to failure probability. Subsequently, fewer number of function evaluations is required compared with existing competitive works. Moreover, the key advantage of the proposed method is that it is a single-loop process, i.e., the selection of training samples, dimension reduction and Kriging modeling have been combined simultaneously at each iteration with adaptive manner. However, the existing competitive methods, generally, have two independent stages, i.e., stage one is collecting training samples and stage two is performing PCA and Kriging modelling for time-dependent reliability analysis. Thus, these methods cannot fully utilize the information provided by PCA to find optimal training samples. It is noteworthy that the dimension reduction is also an adaptive process in the proposed method, i.e., dimension reduction changes with the training samples at each iteration. The applicability, accuracy and efficiency of the proposed method are validated through three numerical examples and one engineering example.

Image-Based Feedback Control Using Tensor Analysis

In manufacturing systems, many quality measurements are in the form of images, including overlay measurements in the semiconductor manufacturing and dimensional deformation profiles of fuselages in an aircraft assembly process. To reduce the process variability and ensure on-target quality, process control strategies should be deployed, in which the high-dimensional image output is controlled by one or more input variables. To design an effective control strategy, the process model should be first estimated via relationship exploration between the image output and inputs, off-line. Next, the control law is formulated by minimizing the control objective function online. The main challenges of achieving such a control strategy include (i) the high dimensional output of a regression model, (ii) the integrated analysis of both the spatial structure of image outputs and the temporal structure of the image sequence, and (iii) non-iid noises. To address these challenges, we propose a novel tensor-based process control approach by incorporating the tensor time series and regression techniques. Based on the process model, we can then obtain the control law by minimizing a control objective function. Although our proposed approach is motivated by a 2D image case, it can be extended to higher-order tensors such as point clouds. Simulation and case studies show that our proposed method is more effective than benchmarks in terms of relative mean square error.

Fully hyperbolic convolutional neural networks

Convolutional neural networks (CNN) have recently seen tremendous success in various computer vision tasks. However, their application to problems with high dimensional input and output, such as high-resolution image and video segmentation or 3D medical imaging, has been limited by various factors. Primarily, in the training stage, it is necessary to store network activations for back-propagation. In these settings, the memory requirements associated with storing activations can exceed what is feasible with current hardware, especially for problems in 3D. Motivated by the propagation of signals over physical networks, that are governed by the hyperbolic Telegraph equation, in this work we introduce a fully conservative hyperbolic network for problems with high-dimensional input and output. We introduce a coarsening operation that allows completely reversible CNNs by using a learnable discrete wavelet transform and its inverse to both coarsen and interpolate the network state and change the number of channels. We show that fully reversible networks are able to achieve results comparable to the state of the art in 4D time-lapse hyper-spectral image segmentation and full 3D video segmentation, with a much lower memory footprint that is a constant independent of the network depth. We also extend the use of such networks to variational auto-encoders, where optimization begins from an exact recovery and we discover the level of compression through optimization.