High-dimensional Input Research Articles

An improved neural operator framework for large-scale CO2 storage operations

Geologic carbon storage (GCS) is a practical solution to mitigate the impact of climate change and achieve net-zero carbon emissions. However, due to complex subsurface characteristics and operational constraints, GCS can potentially induce seismic events and leakage to groundwater resources. Moreover, predicting subsurface response in pressure and saturation changes during GCS requires high-fidelity models with high computational times. Machine learning (ML) offers a promising solution to alleviate these challenges. Yet, ML-driven approaches in subsurface physics face significant obstacles, such as the substantial computational resources required for training large-scale models and the accuracy requirement as a reliable surrogate model. To tackle these issues, we propose an improved neural operator (INO) that has improved a DeepONet architecture to handle a realistic reservoir model efficiently, with computational efficiency and high accuracy.Our proposed INO framework has three key benefits: precision, adaptability, and cost-efficiency. For precision, it achieves an average root mean square error (RMSE) of 0.05% (1.5 psi) compared to the average reservoir pressure (3,150 psi) and a RMSE value of 0.016 in void fraction for CO2 saturation among testing cases. Although it is sensitive to high-error areas with steep state variable gradients, proper domain decomposition, ensemble methods, and/or integrating domain-specific knowledge could improve this limitation. For adaptability, the INO framework can be trained with a subset of the computational domain during each backpropagation to achieve better training adaptability, even for problems with high-dimensional inputs (i.e., heterogeneous fields). For example, it can be trained with random subsampling of only 0.2% of the full domain but can predict pressure and saturation in any space and time within the entire domain. For cost efficiency, training procedures with a subset of the whole computational domain significantly improve computational time. For example, for 90 training cases with 1.73 million cells and 50-time steps, the overall training time was about 2.5 h using a single GPU. A trained INO model takes 1 s per evaluation over 1.73 million and 50 time stamps, much faster than a high-fidelity simulator. Overall, the proposed INO approach with these improved benefits will enable us to perform large-scale GCS applications, enhancing the potential of ML models in the subsurface and other multiphysics problems.

Nonparametric Machine Learning for Stochastic Frontier Analysis: A Bayesian Additive Regression Tree Approach

The stochastic frontier model is widely used in economics, finance, and management to estimate the production function and efficiency of a firm or industry. In the current literature of stochastic frontier analysis, parametric forms of production functions, such as Cobb-Douglas and Translog, are often assumed a priori without validation, which may suffer from model misspecification and lead to biased estimates of efficiency. To address this issue, a new set of stochastic frontier models is proposed, namely, 1) monBART-SFM that constructs a monotone-constrained nonparametric production function via an extension of the monotone Bayesian Additive Regression Tree (monBART) framework; 2) softBART-SFM that is built upon softBART, a smooth version of BART, where both models offer greater flexibility in modeling complex and nonlinear production functions that may have high dimensional inputs. The performance of the proposed method is illustrated through simulation studies and a real data application.

Bayesian Safety Validation for Failure Probability Estimation of Black-Box Systems

Estimating the probability of failure is an important step in the certification of safety-critical systems. Efficient estimation methods are often needed due to the challenges posed by high-dimensional input spaces, risky test scenarios, and computationally expensive simulators. This work frames the problem of black-box safety validation as a Bayesian optimization problem and introduces a method that iteratively fits a probabilistic surrogate model to efficiently predict failures. The algorithm is designed to search for failures, compute the most-likely failure, and estimate the failure probability over an operating domain using importance sampling. We introduce three acquisition functions that aim to reduce uncertainty by covering the design space, optimize the analytically derived failure boundaries, and sample the predicted failure regions. Results show this Bayesian safety validation approach provides a more accurate estimate of failure probability with orders of magnitude fewer samples and performs well across various safety validation metrics. We demonstrate this approach on three test problems, a stochastic decision-making system, and a neural-network-based runway detection system. This work is open-sourced (https://github.com/sisl/BayesianSafetyValidation.jl) and is currently being used to supplement the FAA certification process of the machine learning components for an autonomous cargo aircraft.

Robust Spherical Laplacian Embedding.

Laplacian embedding (LE) aims to project high-dimensional input data samples, which often contain nonlinear structures, into a low-dimensional space. However, existing distance functions used in the embedding space fail to provide discriminative representations for real-world datasets, especially those related to text analysis or image processing. Cosine similarity measurements are advantageous in dealing with sparse data but are fragile to the impact of outliers and noise from samples or data features. In this work, we propose robust spherical LE (RS-LE), a powerful method that builds the LE on a novel metric that unifies the Euclidean distance and cosine similarity in a spherical space using a robust lp,p -norm. We introduce an efficient iterative algorithm, the proximal alternating linearized minimization (ALM) method, to solve the difficult optimization problem that arises from the new metric, which is neither convex nor smooth. This algorithm allows the solution to achieve both global and objective convergence. Our findings are presented in rigorous theoretical analysis and validated in experiments.

Physics-informed generator-encoder adversarial networks with latent space matching for stochastic differential equations

We propose a new class of physics-informed neural networks, called Physics-Informed Generator-Encoder Adversarial Networks, to effectively address the challenges posed by forward, inverse, and mixed problems in stochastic differential equations (SDEs). In these scenarios, while governing equations are known, the available data consist of only a limited set of snapshots for system parameters. Our model consists of two key components: the generator and the encoder, both updated alternately by gradient descent. In contrast to previous approaches that directly match approximated solutions with real snapshots, we employ an indirect matching operating within the lower-dimensional latent feature space. This method circumvents challenges associated with high-dimensional inputs and complex data distributions in solving SDEs. Numerical experiments indicate that, compared to existing deep learning solvers, our proposed approach not only demonstrates superior accuracy but also exhibits advantages in both computational efficiency and model complexity.

A physics and data co-driven surrogate modeling method for high-dimensional rare event simulation

This paper presents a physics and data co-driven surrogate modeling method for efficient rare event simulation of civil and mechanical systems with high-dimensional input uncertainties. The method fuses interpretable low-fidelity physical models with data-driven error corrections. The hypothesis is that a well-designed and well-trained simplified physical model can preserve salient features of the original model, while data-fitting techniques can fill the remaining gaps between the surrogate and original model predictions. The coupled physics-data-driven surrogate model is adaptively trained using active learning, aiming to achieve a high correlation and small bias between the surrogate and original model responses in the critical parametric region of a rare event. A final importance sampling step is introduced to correct the surrogate model-based probability estimations. Static and dynamic problems with input uncertainties modeled by random field and stochastic process are studied to demonstrate the proposed method.

A clustering-based partially stratified sampling for high-dimensional structural reliability assessment

Assessing structural reliability problem with high-dimensional random inputs is still challenging due to the “curse of dimensionality”. In this paper, this challenge is addressed by extending the Generalized Distribution Reconstruction Method via Characteristic Function Inversion (GDRM-CFI). Specifically, a clustering-based partially stratified sampling method is proposed for selecting high-dimensional points to numerically evaluate the characteristic function (CF) curve of complex high-dimensional problems. An improved number-theoretical method (i-NTM) is used to establish a uniform, efficient point set, ensuring determinism and reducing variability. Subsequently, a partial stratification approach partitions the high-dimensional space into orthogonal two-dimensional subspaces. The fundamental point set is projected into each subspace, and the k-means clustering algorithm identifies centroids within each, acting as representative points. The complete set of representative points from all subspaces formulates the high-dimensional point set. Numerical examples are investigated, which demonstrate the proposed method is effective for high-dimensional structural reliability assessment.

Real-time prediction of fuel consumption and emissions based on deep autoencoding support vector regression for cylinder pressure-based feedback control of marine diesel engines

Predictive models serving as virtual sensors for online optimization and feedback control of diesel engines is gaining increasing attention. However, existing prediction models fall short in simultaneously achieving high prediction accuracy and fast computational speed. In this paper, a novel machine learning algorithm called deep autoencoding support vector regression (DASVR) was proposed, which combines the powerful non-linear feature extraction capability of artificial neural network (ANN) with the good adaptability of support vector regression (SVR) to low-dimensional input spaces. ANN-based autoencoder is firstly employed to extract features from the original high-dimensional input space, forming a low-dimensional latent variable space. SVR is then employed to perform non-linear mapping between latent variables and target output variables. Experiments were conducted on a 16-cylinder marine diesel engine under different load, rail pressure, and injection timing conditions. Prediction models for fuel consumption, NOx, PM, HC, and PM emissions were established based on experimental data and DASVR algorithm. The maximum error of DASVR-based models for the five desired output variables is less than 3.8 % under different operating conditions. DAVSR-based predictive models outperform conventional ANN-based and SVR-based models in terms of both prediction accuracy and real-time capability, and can theoretically meet the real-time requirement below 3000 r/min.

Deep learning-based predictive models for laser direct drive at the Omega Laser Facility

The rich and complex physics of inertial confinement fusion provides a unique and challenging space for high-fidelity first-principles modeling. Consequently, simulation codes that are used to design experiments are computationally expensive and lack the predictive capability required for extensive parameter exploration in search of a high-performing design for laser direct drive. In this article, we present two deep-learning-based predictive models intended to address these difficulties. The first model (TL DNN) acts as a fast emulator of simulations as well as experiments at the Omega Laser Facility. This model is trained on a simulation database and subsequently calibrated on experimental data using transfer learning. To facilitate the development of this model, an autoencoder is developed to reduce the dimensionality of the input space by compressing the laser pulse input. The model predicts key experimental scalar observables of Omega experiments with high accuracy and minimal computational cost. This deep neural net enables rapid exploration of a high-dimensional input parameter space for an optimal implosion design. The second model (DNN SM+) aims to extend the statistical modeling work of Lees et al. [Phys. Rev. Lett. 127, 105001 (2021)], by increasing the complexity of the model space and allowing for coupling between degradation terms. Since the model capacity of DNN SM+ is higher than the model of Lees et al., DNN SM+ can potentially provide an improvement in predictive capability, and we use this model to provide insight into complicated degradation dependencies.

Spatial-temporal self-attention network based on bayesian optimization for light olefins yields prediction in methanol-to-olefins process

Methanol-to-olefins (MTO), as an alternative pathway for the synthesis of light olefins (ethylene and propylene), has gained extensive attention. Accurate prediction of light olefins yields can effectively facilitate process monitoring and optimization, as they are significant economic indexes and stable operation indicators of the industrial MTO process. However, the nonlinearity and dynamic interactions among process variables pose challenges for the prediction using traditional statistical methods. Additionally, physical-based methods relying on first-principle theory are always limited by an insufficient understanding of reaction mechanisms. In contrast, data-driven methods offer a viable solution for the prediction based solely on process data without requiring extensive process knowledge. Therefore, in this work, a data-driven approach that integrates spatial and temporal self-attention modules is proposed to capture complex interactions. Furthermore, Bayesian optimization is employed to determine the optimum hyperparameters and enhance the accuracy of the model. Studies on an actual MTO process demonstrate the superior prediction performance of the proposed model compared to baseline models. Specifically, 24 process variables are selected as the high-dimensional inputs, and yields of ethylene and propylene, as the low-dimensional outputs, are successfully predicted at various prediction horizons ranging from 2 to 8 h.

RETROFIT: Real-Time Control of Time-Dependent 3D Point Cloud Profiles

Abstract In modern manufacturing processes, ensuring the precision of 3D profiles of products is crucial. Nonetheless, achieving this accuracy is challenging due to the complex interactions between process inputs and the data structure of the 3D profile data. Our solution, a 3D profile-based control framework, addresses this challenge by actively adapting and controlling the manufacturing process to enhance 3D shape accuracy. 3D profile scans represent the ultimate measure of desired part quality. Therefore, utilizing them as the system responses for control purposes yields the most direct and effective feedback. We leverage recent advancements from Koopman operator theory to create an effective model-based control strategy. Initially, we estimate the process model by exploring the relationship between 3D profiles and heterogeneous process inputs. Then, we formulate an online model predictive control law. Challenges include dealing with unstructured, high-dimensional 3D point cloud data, capturing spatial and temporal structures, and integrating heterogeneous, high-dimensional process input data into the control model. To overcome these challenges, we introduce RETROFIT, a solution designed for the real-time control of time-dependent 3D point cloud profiles. Unlike traditional models, RETROFIT is not bound by linear assumptions and can handle unstructured 3D point cloud data directly. We demonstrate its effectiveness through a wire arc additive manufacturing case study, highlighting its potential to enhance 3D profile accuracy in manufacturing processes.

Assessing input parameter hyperspace and parameter identifiability in a cardiovascular system model via sensitivity analysis

We aim to clarify our understanding of the process of state-space model input parameter identification, known, within the clinical context, as model personalisation. To do so, we apply reference sensitivity and identifiability techniques to a lumped parameter, single ventricle representation of the systemic circulation, chosen in view of its relative simplicity and prior art. We attempt to quantify the reliability of input parameter identifiability through the lens of 4 clinically relevant measurements and the attendant difficulty in personalising the model. In turn, this that we extend existing methods which combine both parameter influence and orthogonality, to global sensitivities. By examining different parameter sensitivity evaluation methodologies, we investigate the stability of optimal parameter subsets which are commonly used to aid clinical investigations. In order to perform the personalisation process, one must understand the complexity of the high dimensional input parameter hyperspace associated with this class of model. By utilising Sobol indices, we propose a domain-agnostic and intuitive approach. This involves varying the bounds of the input parameter space relative to the model’s base state. These investigations yield a pseudo-mapping of the input hyperspace, cementing our understanding of the role of identifiable input parameters in the state-space model. Our findings suggest a novel global methodology for input parameter identifiability and input hyperspace mapping, providing valuable insights into solving the personalisation process.

Inverse design of two-function transmission-type reconfigurable polarization control metasurfaces based on deep learning

Compared with single-function metasurfaces, the design difficulty of multi-function metasurfaces increases significantly. This paper introduces an inverse design method based on deep learning to address this challenge. By this method, a transmission-type reconfigurable polarization control metasurface (TRPCM) with two functions is rapidly designed. The network model used in the method consists of an electromagnetic parameter reconstruction network model and an inverse prediction network model. The combination of the two models can solve the problem of difficulty in defining high-dimensional inputs in traditional inverse design, and achieve accurate prediction of metasurface structure parameters under given design targets. To optimize the hyperparameters of the neural network model, a genetic algorithm was introduced. To solve the non-uniqueness problem of inverse design, a method for eliminating similar data by calculating Euclidean Distance was introduced. Both schemes further improve the predictive performance of the proposed network model. Finally, six design targets were set based on the TRPCM. The structural parameters of the metasurface were successfully predicted using two neural network models and achieved the required performance. On this basis, a set of parameters was selected for experimental validation. By controlling the ON or OFF of the PIN diodes, the fabricated metasurface achieves two functions: linear-to-circular polarization conversion and linear polarization maintenance in the range of 2–3.6 GHz. Study results show that the inverse design scheme proposed in the paper is feasible and practical for solving the rapid optimization design of complex multi-function metasurfaces.

An encoder-decoder ConvLSTM surrogate model for simulating geological CO2 sequestration with dynamic well controls

In Geological Carbon Sequestration (GCS), effectively managing the project requires predicting state variables such as pressure and saturation. However, numerical simulation of multiphase flow in subsurface porous media involves solving large linear algebra systems, resulting in a substantial computational burden. This can make it impractical for real-time history matching or optimization. Meanwhile, deep learning-based surrogate models are emerging as fast and accurate approximators. This study delves into the application of the Encoder-Decoder Convolutional Long Short-Term Memory (ED-ConvLSTM) neural network for predicting the complex evolution of state variables under dynamic CO2 injection schemes. Inception blocks enhanced with light-weighted attention modules are introduced in the encoder to extract high-dimensional input features. ConvLSTM is employed to propagate spatial temporal information in the low-dimensional latent space. Further, progressive upsampling blocks are used to reconstruct the latent features for the desired output. Instead of taking discrete time steps as an input feature, the proposed network captures the dynamic dependencies with the inherent ConvLSTM cell. The network has access to data at only portion of the initial time steps during training stage, while it is used to predict the state variables at unseen time steps during testing stage. Results show that the network can produce excellent predictions for both pressure and saturation, even at unseen future time steps. The remarkable generalizability to different geological permeability fields is also evaluated. ED-ConvLSTM outperforms the standard U-Net by far, especially when predicting beyond the training time period. These numerical experiments demonstrate the advantages of ED-ConvLSTM in terms of prediction accuracy, extrapolability and generalizability. This study highlights the importance of incorporating recurrent connections into the deep neural networks for simulating time-dependent multiphase flow problems. The proposed methodology has great potential in GCS surrogate modeling and offers a possible approach for real-time optimization of CO2 injection.

Development and trending of deep learning methods for wind power predictions

With the increasing data availability in wind power production processes due to advanced sensing technologies, data-driven models have become prevalent in studying wind power prediction (WPP) methods. Deep learning models have gained popularity in recent years due to their ability of handling high-dimensional input, automating data feature engineering, and providing high flexibility in modeling. However, with a large volume of deep learning based WPP studies developed in recent literature, it is important to survey the existing developments and their contributions in solving the issue of wind power uncertainty. This paper revisits deep learning-based wind power prediction studies from two perspectives, deep learning-enabled WPP formulations and developed deep learning methods. The advancement of WPP formulations is summarized from the following perspectives, the considered input and output designs as well as the performance evaluation metrics. The technical aspect review of deep learning leveraged in WPPs focuses on its advancement in feature processing and prediction model development. To derive a more insightful conclusion on the so-far development, over 140 recent deep learning-based WPP studies have been covered. Meanwhile, we have also conducted a comparative study on a set of deep models widely used in WPP studies and recently developed in the machine learning community. Results show that DLinear obtains more than 2% improvements by benchmarking a set of strong deep learning models. Potential research directions for WPPs, which can bring profound impacts, are also highlighted.

Predicting anomalies in computer networks using autoencoder-based representation learning

<span>Recent improvements in the internet of things (IoT), cloud services, and network data variety have increased the demand for complex anomaly detection algorithms in network intrusion detection systems (IDSs) capable of dealing with sophisticated network threats. Academics are interested in deep and machine learning (ML) breakthroughs because they have the potential to address complex challenges such as zero-day attacks. In comparison to firewalls, IDS are the initial line of network security. This study suggests merging supervised and unsupervised learning in identification systems IDS. Support vector machine (SVM) is an anomaly-based classification classifier. Deep autoencoder (DAE) lowers dimensionality. DAE are compared to principal component analysis (PCA) in this study, and hyper-parameters for F-1 micro score and balanced accuracy are specified. We have an uneven set of data classes. precision-recall curves, average precision (AP) score, train-test times, t-SNE, grid search, and L1/L2 regularization methods are used. KDDTrain+ and KDDTest+ datasets will be used in our model. For classification and performance, the DAE+SVM neural network technique is successful. Autoencoders outperformed linear PCA in terms of capturing valuable input attributes using t-SNE to embed high dimensional inputs on a two-dimensional plane. Our neural system outperforms solo SVM and PCA encoded SVM in multi-class scenarios.</span>

Interpretable machine learning approach for exploring process-structure-property relationships in metal additive manufacturing

Process-structure-property (PSP) relationships are critical to the optimization of manufacturing processes, but establishing these relationships typically involves time- and cost- consuming experiments, especially for additive manufacturing (AM) due to the large number of process parameters involved. In this study, we develop a novel and interpretable machine learning approach for predicting, optimizing, and expanding the process window of laser powder bed fusion (LPBF) while simultaneously establishing PSP relationships, using AlSi10Mg as an example. Our iterative, error-targeted method substantially decreases the amount of experimentation required. Gaussian process regression (GPR) was employed as the predictive model, incorporating multiple input variables (e.g., process parameters, relative density, melt pool morphology, cellular structure, and grain structure), for predicting three mechanical properties (i.e., yield strength, ultimate tensile strength, and % elongation). A comparison of model predictions and experimental data outside the training scope reveals that the prediction accuracy can be improved with higher dimensional inputs and further enhanced by a multi-output model that accounts for correlations between the different mechanical properties. Additionally, the GPR kernel’s hyperparameter for each input enables feature selection and model interpretability. The proposed approach can assist with finding the most critical variables affecting mechanical performance, establishing the PSP relationships of AM fabricated alloys, and providing guidance for tailoring the final properties. The methodology presented in this study can be applied to various AM techniques and materials to broaden the process window to achieve previously unattainable mechanical properties, as well as to gain a deeper understanding of the PSP relationships.

An Improved Machine Learning Model for Pure Component Property Estimation

Information on the physicochemical properties of chemical species is an important prerequisite when performing tasks such as process design and product design. However, the lack of extensive data and high experimental costs hinder the development of prediction techniques for these properties. Moreover, accuracy and predictive capabilities still limit the scope and applicability of most property estimation methods. This paper proposes a new Gaussian process-based modeling framework that aims to manage a discrete and high-dimensional input space related to molecular structure representation with the group-contribution approach. A warping function is used to map discrete input into a continuous domain in order to adjust the correlation between different compounds. Prior selection techniques, including prior elicitation and prior predictive checking, are also applied during the building procedure to provide the model with more information from previous research findings. The framework is assessed using datasets of varying sizes for 20 pure component properties. For 18 out of the 20 pure component properties, the new models are found to give improved accuracy and predictive power in comparison with other published models, with and without machine learning.

Deep Reinforcement Learning: A Survey.

Deep reinforcement learning (DRL) integrates the feature representation ability of deep learning with the decision-making ability of reinforcement learning so that it can achieve powerful end-to-end learning control capabilities. In the past decade, DRL has made substantial advances in many tasks that require perceiving high-dimensional input and making optimal or near-optimal decisions. However, there are still many challenging problems in the theory and applications of DRL, especially in learning control tasks with limited samples, sparse rewards, and multiple agents. Researchers have proposed various solutions and new theories to solve these problems and promote the development of DRL. In addition, deep learning has stimulated the further development of many subfields of reinforcement learning, such as hierarchical reinforcement learning (HRL), multiagent reinforcement learning, and imitation learning. This article gives a comprehensive overview of the fundamental theories, key algorithms, and primary research domains of DRL. In addition to value-based and policy-based DRL algorithms, the advances in maximum entropy-based DRL are summarized. The future research topics of DRL are also analyzed and discussed.

Spatio-temporal water height prediction for dam break flows using deep learning

Spatio-temporal prediction of the water height for dam break flow is of great significance in flood control projects. In this article, we propose a novel deep learning model named AE-LSTM-ATT that combines AutoEncoder (AE) and Long Short-Term Memory with Attention (LSTM-ATT) to predict the dynamic behavior of the dam-break water height field in spatial and temporal dimensions. The prediction performance of the AE-LSTM-ATT model was demonstrated using dam-break flow cases with different initial heights of the water column. The AE first compresses high-dimensional inputs into low-dimensional latent space using its encoder to enable a fast computation process. Then, the latent space is adopted as an input for the LSTM-ATT to predict low-dimensional representations at future time instances that can be restored to the water height field by the decoder of the AE. The major findings of this article include three parts: (1) AE-LSTM-ATT model can successfully achieve the spatio-temporal prediction of the water height for dam break flows with high precision; (2) The proposed AE presents much better performance than those of the widely used Principal Component Analysis (PCA) and Kernel Principal Component Analysis (KPCA) in the dimension reduction; (3) The developed LSTM-ATT performs far better than the commonly used Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) in the time-series prediction. These promising results suggest that the proposed AE-LSTM-ATT model can play an essential role in capturing and advancing the spatio-temporal features of dam-break water height, which can be applied in the field of real-time decision making for dam-break floods.