High-dimensional Input Space Research Articles

A nested non-intrusive stochastic isogeometric method for nonlinear thermal vibration of FGM plates under random loading

This paper introduces an adaptive nested non-invasive dynamic SIGA method based on RBFNN for the nonlinear thermal vibration analysis of FGM plates under random loading. This method is robust and accurate in high-dimensional input spaces regardless of the sample sequence, addresses the low convergence rate of QMCS, and solves the stochastic dynamic response probability density function under random loading. Firstly, the multidimensional input space for random loading is constructed based on several mutually independent random variables via the stationary Gaussian stochastic process simulation technique. The nonlinear FGM plate model subjected to random loading in the thermal environment is modeled using HSDT with von Kármán strain-displacement relation within the IGA framework. Subsequently, a nested non-invasive dynamic response surface analysis method, SIGA-RBFNN, is proposed based on the RBFNN technique. Analysis of the FGM plate response under random loading demonstrates that SIGA-RBFNN, compared to tensor-product-based SIGA methods, is less affected by sample sequence types. It effectively addresses high-dimensional stochasticity and offers significant advantages in robustness, accuracy, and efficiency. Finally, SIGA-RBFNN compensates for the low convergence speed of QMCS and successfully solves the FGM plate displacement response statistics and probability density function under random loading.

SpaDE: Semantic Locality Preserving Biclustering for Neuroimaging Data.

The most discriminative and revealing patterns in the neuroimaging population are often confined to smaller subdivisions of the samples and features. Especially in neuropsychiatric conditions, symptoms are expressed within micro subgroups of individuals and may only underly a subset of neurological mechanisms. As such, running a whole-population analysis yields suboptimal outcomes leading to reduced specificity and interpretability. Biclustering is a potential solution since subject heterogeneity makes one-dimensional clustering less effective in this realm. Yet, high dimensional sparse input space and semantically incoherent grouping of attributes make post hoc analysis challenging. Therefore, we propose a deep neural network called semantic locality preserving auto decoder (SpaDE), for unsupervised feature learning and biclustering. SpaDE produces coherent subgroups of subjects and neural features preserving semantic locality and enhancing neurobiological interpretability. Also, it regularizes for sparsity to improve representation learning. We employ SpaDE on human brain connectome collected from schizophrenia (SZ) and healthy control (HC) subjects. The model outperforms several state-of-the-art biclustering methods. Our method extracts modular neural communities showing significant (HC/SZ) group differences in distinct brain networks including visual, sensorimotor, and subcortical. Moreover, these bi-clustered connectivity substructures exhibit substantial relations with various cognitive measures such as attention, working memory, and visual learning.

Bayesian Safety Validation for Failure Probability Estimation of Black-Box Systems

Estimating the probability of failure is an important step in the certification of safety-critical systems. Efficient estimation methods are often needed due to the challenges posed by high-dimensional input spaces, risky test scenarios, and computationally expensive simulators. This work frames the problem of black-box safety validation as a Bayesian optimization problem and introduces a method that iteratively fits a probabilistic surrogate model to efficiently predict failures. The algorithm is designed to search for failures, compute the most-likely failure, and estimate the failure probability over an operating domain using importance sampling. We introduce three acquisition functions that aim to reduce uncertainty by covering the design space, optimize the analytically derived failure boundaries, and sample the predicted failure regions. Results show this Bayesian safety validation approach provides a more accurate estimate of failure probability with orders of magnitude fewer samples and performs well across various safety validation metrics. We demonstrate this approach on three test problems, a stochastic decision-making system, and a neural-network-based runway detection system. This work is open-sourced (https://github.com/sisl/BayesianSafetyValidation.jl) and is currently being used to supplement the FAA certification process of the machine learning components for an autonomous cargo aircraft.

Real-time prediction of fuel consumption and emissions based on deep autoencoding support vector regression for cylinder pressure-based feedback control of marine diesel engines

Predictive models serving as virtual sensors for online optimization and feedback control of diesel engines is gaining increasing attention. However, existing prediction models fall short in simultaneously achieving high prediction accuracy and fast computational speed. In this paper, a novel machine learning algorithm called deep autoencoding support vector regression (DASVR) was proposed, which combines the powerful non-linear feature extraction capability of artificial neural network (ANN) with the good adaptability of support vector regression (SVR) to low-dimensional input spaces. ANN-based autoencoder is firstly employed to extract features from the original high-dimensional input space, forming a low-dimensional latent variable space. SVR is then employed to perform non-linear mapping between latent variables and target output variables. Experiments were conducted on a 16-cylinder marine diesel engine under different load, rail pressure, and injection timing conditions. Prediction models for fuel consumption, NOx, PM, HC, and PM emissions were established based on experimental data and DASVR algorithm. The maximum error of DASVR-based models for the five desired output variables is less than 3.8 % under different operating conditions. DAVSR-based predictive models outperform conventional ANN-based and SVR-based models in terms of both prediction accuracy and real-time capability, and can theoretically meet the real-time requirement below 3000 r/min.

An Improved Machine Learning Model for Pure Component Property Estimation

Information on the physicochemical properties of chemical species is an important prerequisite when performing tasks such as process design and product design. However, the lack of extensive data and high experimental costs hinder the development of prediction techniques for these properties. Moreover, accuracy and predictive capabilities still limit the scope and applicability of most property estimation methods. This paper proposes a new Gaussian process-based modeling framework that aims to manage a discrete and high-dimensional input space related to molecular structure representation with the group-contribution approach. A warping function is used to map discrete input into a continuous domain in order to adjust the correlation between different compounds. Prior selection techniques, including prior elicitation and prior predictive checking, are also applied during the building procedure to provide the model with more information from previous research findings. The framework is assessed using datasets of varying sizes for 20 pure component properties. For 18 out of the 20 pure component properties, the new models are found to give improved accuracy and predictive power in comparison with other published models, with and without machine learning.

Deep Reinforcement Learning Unleashing the Power of AI in Decision-Making

Deep Reinforcement Learning (DRL) has emerged as a transformative paradigm in the field of artificial intelligence (AI), offering unprecedented capabilities in decision-making across diverse domains. This article explores the profound impact of DRL on enhancing the decision-making capabilities of AI systems, elucidating its underlying principles, applications, and implications.DRL represents a fusion of deep learning and reinforcement learning, enabling machines to learn complex behaviors and make decisions by interacting with their environment. The utilization of neural networks allows DRL algorithms to handle high-dimensional input spaces, making it well-suited for tasks that involve intricate decision-making processes.One of the key strengths of DRL lies in its ability to address problems with sparse and delayed rewards, common challenges in traditional reinforcement learning. Through a process of trial and error, DRL algorithms can learn optimal decision strategies by navigating through a vast decision space, adapting to dynamic environments, and maximizing cumulative rewards over time.The applications of DRL span various domains, including robotics, finance, healthcare, gaming, and autonomous systems. In robotics, DRL facilitates the development of intelligent agents capable of autonomously navigating complex environments, performing intricate tasks, and adapting to unforeseen circumstances. In finance, DRL is leveraged for portfolio optimization, algorithmic trading, and risk management, demonstrating its potential to revolutionize traditional financial strategies.

ANOVA-GP Modeling for High-Dimensional Bayesian Inverse Problems

Markov chain Monte Carlo (MCMC) stands out as an effective method for tackling Bayesian inverse problems. However, when dealing with computationally expensive forward models and high-dimensional parameter spaces, the challenge of repeated sampling becomes pronounced. A common strategy to address this challenge is to construct an inexpensive surrogate of the forward model, which cuts the computational cost of individual samples. While the Gaussian process (GP) is widely used as a surrogate modeling strategy, its applicability can be limited when dealing with high-dimensional input or output spaces. This paper presents a novel approach that combines the analysis of variance (ANOVA) decomposition method with Gaussian process regression to handle high-dimensional Bayesian inverse problems. Initially, the ANOVA method is employed to reduce the dimension of the parameter space, which decomposes the original high-dimensional problem into several low-dimensional sub-problems. Subsequently, principal component analysis (PCA) is utilized to reduce the dimension of the output space on each sub-problem. Finally, a Gaussian process model with a low-dimensional input and output is constructed for each sub-problem. In addition to this methodology, an adaptive ANOVA-GP-MCMC algorithm is proposed, which further enhances the adaptability and efficiency of the method in the Bayesian inversion setting. The accuracy and computational efficiency of the proposed approach are validated through numerical experiments. This innovative integration of ANOVA and Gaussian processes provides a promising solution to address challenges associated with high-dimensional parameter spaces and computationally expensive forward models in Bayesian inference.

Mapping top-two-floor corner coordinates to building strains in deep latent space

Structural health monitoring is vital for ensuring building safety but faces numerous challenges, such as sensor durability, maintenance costs, and data loss. To address these issues, this study introduces a deep learning framework designed to obviate the need for ongoing maintenance of strain gauges. The framework aims to map the top-two-floor corner coordinates of a building to its strains in deep latent space. The framework comprises four steps: grouping coordinates, standardizing coordinates, generating a coordinate map, and learning the deep latent space. Here, learning the deep latent space refers to compressing high-dimensional input space features into low-dimensional deep latent space features to address the mapping problem while reducing overfitting and improving the explainability of a deep neural network for practical engineering applications. Once the deep latent space is learned, building strains can be predicted by observing only the top part of the building, eliminating the need for continuous maintenance of strain gauges in structural health monitoring. The proposed framework is evaluated on a four-story scaled building and demonstrates exceptional mapping performance in the learned deep latent space.

Place-and-Route Analysis of FPGA Implementation of Nested Hardware Self-Organizing Map Architecture

Self-organizing map (SOM) is a type of artificial neural network that provides a nonlinear mapping from a given high-dimensional input space to a low-dimensional map of neurons for clustering. The clustering of high-dimensional vectors is too slow for SOM when implemented in software. In such cases, an application-specific hardware SOM accelerator is highly desirable. Field-programmable gate array (FPGA) implementation is a popular platform to implement hardware SOM. In our previous work, a nested hardware SOM architecture, which has a homogeneous modular structure to enhance expandability, was proposed. This paper investigates the impact of the nested hardware SOM on FPGA implementation tools that perform logic synthesis, place, and route (PAR). Experiments revealed that the nested architecture provided better results in resource usage and performance. FPGA resource usage of the nested architecture was 97.2% of that of the flat design on average. Importantly, the nested architecture operated at 10% higher clock frequencies compared to flat SOM designs. In addition, the pipeline computation was improved by increasing the pipeline stages so that it operates with a higher clock frequency. The operable clock frequency was 81 MHz, which was 21 MHz higher than its predecessor.

Bayesian optimization approach to quantify the effect of input parameter uncertainty on predictions of numerical physics simulations

An understanding of how input parameter uncertainty in the numerical simulation of physical models leads to simulation output uncertainty is a challenging task. Common methods for quantifying output uncertainty, such as performing a grid or random search over the model input space, are computationally intractable for a large number of input parameters represented by a high-dimensional input space. It is, therefore, generally unclear as to whether a numerical simulation can reproduce a particular outcome (e.g., a set of experimental results) with a plausible set of model input parameters. Here, we present a method for efficiently searching the input space using Bayesian optimization to minimize the difference between the simulation output and a set of experimental results. Our method allows explicit evaluation of the probability that the simulation can reproduce the measured experimental results in the region of input space defined by the uncertainty in each input parameter. We apply this method to the simulation of charge-carrier dynamics in the perovskite semiconductor methyl-ammonium lead iodide (MAPbI3), which has attracted attention as a light harvesting material in solar cells. From our analysis, we conclude that the formation of large polarons, quasiparticles created by the coupling of excess electrons or holes with ionic vibrations, cannot explain the experimentally observed temperature dependence of electron mobility.

Uncertainty Quantification for Thermodynamic Simulations with High-Dimensional Input Spaces Using Sparse Polynomial Chaos Expansion: Retrofit of a Large Thermal Power Plant

The assessment of the future thermodynamics performance of a retrofitted heat and power production unit is prone to many uncertainties due to the large number of parameters involved in the modeling of all its components. To carry out uncertainty quantification analysis, alternatives to the traditional Monte Carlo method must be used due to the large stochastic dimension of the problem. In this paper, sparse polynomial chaos expansion (SPCE) is applied to the retrofit of a large coal-fired power plant into a biomass-fired combined heat and power unit to quantify the main drivers and the overall uncertainty on the plant’s performance. The thermodynamic model encompasses over 180 components and 1500 parameters. A methodology combining the use of SPCE and expert judgment is proposed to narrow down the sources of uncertainty and deliver reliable probability distributions for the main key performance indicators (KPIs). The impact of the uncertainties on each input parameter vary with the considered KPI and its assessment through the computation of Sobol’ indices. For both coal and biomass operations, the most impactful input parameters are the composition of the fuel and its heating value. The uncertainty on the performance and steam quality parameters is not much affected by the retrofit. Key furnace parameters exhibit a skewed probability distribution with large uncertainties, which is a strong attention point in terms of boiler operation and maintenance.

Rolling bearing fault diagnosis under variable working conditions using deep convolutional fuzzy system

This paper addresses the application of a deep convolutional fuzzy system (DCFS) for the fault diagnosis of rolling element bearings. The limitations of the conventional deep convolutional neural network (CNN) are the huge computational load of training the tones of parameters and the lack of interpretability for the corresponding parameters. In this paper, a DCFS-based bearing fault diagnosis method under variable working conditions is proposed. The DCFS on a high dimensional input space is a multilayer connection of many low dimensional fuzzy systems, which can overcome the computational and interpretability problems of the traditional CNN. Moreover, to improve the identification efficiency and diagnosis accuracy, the infinite feature selection (Inf-FS) algorithm is employed to select the most informative fault features. The proposed approach is experimentally demonstrated to be able to identify the different fault types and fault severities of rolling bearings under variable running states.

Multi-filter semi-supervised transformer model for fault diagnosis

Dissolved Gas Analysis (DGA) is the most commonly used method for power transformer fault diagnosis. However, very few reliable and labeled fault DGA samples are available in the transformer substation whilst DGA data without labels is easier to obtain, which makes it difficult to train fault detectors in high-dimensional input space or select features using wrapper methods. Therefore, in order to improve the fault diagnosis accuracy using limited labeled DGA samples but more unlabeled DGA data, this paper proposes a novel multi-filter semi-supervised feature selection method for selecting optimal DGA features and building effective fault diagnosis models. A confidence criterion is also proposed for selecting high confidence unlabeled data to expand the training data set. Five filter techniques based on different evaluation criteria are employed to rank input DGA features, and a feature combination method is then applied to aggregate feature ranks by multiple filters and form a lower-dimensional candidate feature subset. The proposed method has been tested by using the IEC T10 dataset and compared with traditional supervised diagnostic models. The results show that the proposed method works well in optimizing DGA features and improving fault diagnosis accuracy significantly. Besides, the robustness of the selection of optimal feature subset is validated by testing DGA samples from the local power utility.

Data-driven control of agent-based models: An Equation/Variable-free machine learning approach

We present an Equation/Variable free machine learning (EVFML) framework for the control of the collective dynamics of complex/multiscale systems modeled via microscopic/agent-based simulators. The approach obviates the need for construction of surrogate, reduced-order models. The proposed implementation consists of three steps: (A) from high-dimensional agent-based simulations, machine learning, in particular, non-linear manifold learning (Diffusion Maps (DMs)) identifies a set of coarse-grained variables that parametrize the low-dimensional manifold on which the emergent/collective dynamics evolve. The out-of-sample extension and pre-image problems, i.e. the construction of non-linear mappings from the high-dimensional input space to the low-dimensional manifold and back, are solved by coupling DMs with the Nyström extension and Geometric Harmonics, respectively; (B) having identified the manifold and its coordinates, we exploit the Equation-free approach to perform numerical bifurcation analysis of the emergent dynamics; then (C) based on the previous steps, we design data-driven embedded wash-out controllers that drive the agent-based simulators to their intrinsic, imprecisely known, emergent open-loop unstable steady-states, thus demonstrating that the scheme is robust against numerical approximation errors and modeling uncertainty. The efficiency of the framework is illustrated by controlling emergent unstable (i) traveling waves of a deterministic agent-based model of traffic dynamics, and (ii) equilibria of a stochastic financial market model with mimesis.

PnP Operations to Maintain Persistent Feasibility in Linear Constrained Systems

Multi-agent systems often have high dimensional state and input spaces that change rapidly as agents join and leave the system. The plug-and-play (PnP) design paradigm provides a framework to design controllers in this context by shifting most computations for individual agents to design-time. Smaller, numerically tractable PnP algorithms are then used to allow the agents to join and leave the network while maintaining persistent feasibility and stability. In constrained systems, persistent feasibility is assured within feasible control invariant (CI) sets. This work develops a PnP framework to compute these sets for a class of distributed, non-cooperative linear systems. The agents are allowed a preview of the cross-disturbance but no other restrictions are placed on the agents’ controllers. At design-time, undisturbed and maximally disturbed CI sets are found for each agent individually. As the network changes, the agents’ current CI sets are computed using linear combinations of these extreme CI sets. With this, on-line constraint adjustments can be accomplished through distributed optimization problems that scale well for large networks. The overall solution is agnostic to the local controller design and accounts for the local states at the time of the PnP operations.

Reliable emulation of complex functionals by active learning with error control.

A statistical emulator can be used as a surrogate of complex physics-based calculations to drastically reduce the computational cost. Its successful implementation hinges on an accurate representation of the nonlinear response surface with a high-dimensional input space. Conventional "space-filling" designs, including random sampling and Latin hypercube sampling, become inefficient as the dimensionality of the input variables increases, and the predictive accuracy of the emulator can degrade substantially for a test input distant from the training input set. To address this fundamental challenge, we develop a reliable emulator for predicting complex functionals by active learning with error control (ALEC). The algorithm is applicable to infinite-dimensional mapping with high-fidelity predictions and a controlled predictive error. The computational efficiency has been demonstrated by emulating the classical density functional theory (cDFT) calculations, a statistical-mechanical method widely used in modeling the equilibrium properties of complex molecular systems. We show that ALEC is much more accurate than conventional emulators based on the Gaussian processes with "space-filling" designs and alternative active learning methods. In addition, it is computationally more efficient than direct cDFT calculations. ALEC can be a reliable building block for emulating expensive functionals owing to its minimal computational cost, controllable predictive error, and fully automatic features.

Query-Efficient Black-Box Adversarial Attacks Guided by a Transfer-Based Prior.

Adversarial attacks have been extensively studied in recent years since they can identify the vulnerability of deep learning models before deployed. In this paper, we consider the black-box adversarial setting, where the adversary needs to craft adversarial examples without access to the gradients of a target model. Previous methods attempted to approximate the true gradient either by using the transfer gradient of a surrogate white-box model or based on the feedback of model queries. However, the existing methods inevitably suffer from low attack success rates or poor query efficiency since it is difficult to estimate the gradient in a high-dimensional input space with limited information. To address these problems and improve black-box attacks, we propose two prior-guided random gradient-free (PRGF) algorithms based on biased sampling and gradient averaging, respectively. Our methods can take the advantage of a transfer-based prior given by the gradient of a surrogate model and the query information simultaneously. Through theoretical analyses, the transfer-based prior is appropriately integrated with model queries by an optimal coefficient in each method. Extensive experiments demonstrate that, in comparison with the alternative state-of-the-arts, both of our methods require much fewer queries to attack black-box models with higher success rates.

Pi VAE: a stochastic process prior for Bayesian deep learning with MCMC

Stochastic processes provide a mathematically elegant way to model complex data. In theory, they provide flexible priors over function classes that can encode a wide range of interesting assumptions. However, in practice efficient inference by optimisation or marginalisation is difficult, a problem further exacerbated with big data and high dimensional input spaces. We propose a novel variational autoencoder (VAE) called the prior encoding variational autoencoder (pi VAE). pi VAE is a new continuous stochastic process. We use pi VAE to learn low dimensional embeddings of function classes by combining a trainable feature mapping with generative model using a VAE. We show that our framework can accurately learn expressive function classes such as Gaussian processes, but also properties of functions such as their integrals. For popular tasks, such as spatial interpolation, pi VAE achieves state-of-the-art performance both in terms of accuracy and computational efficiency. Perhaps most usefully, we demonstrate an elegant and scalable means of performing fully Bayesian inference for stochastic processes within probabilistic programming languages such as Stan.

Verification of Image-Based Neural Network Controllers Using Generative Models

Although neural networks are effective tools for processing information from image-based sensors to produce control actions, their complex nature limits their use in safety-critical systems. For this reason, recent work has focused on combining techniques in formal methods and reachability analysis to obtain guarantees on the closed-loop performance of neural network controllers. However, these techniques do not scale to the high-dimensional and complicated input space of image-based neural network controllers. This work proposes a method to address these challenges by training a generative adversarial network to map states to plausible input images. Concatenating the generator network with the control network results in a network with a low-dimensional input space, which allows for the use of existing closed-loop verification tools to obtain formal guarantees on the performance of image-based controllers. This approach is applied to provide safety guarantees for an image-based neural network controller for an autonomous aircraft taxi problem. The resulting guarantees are with respect to the set of input images modeled by the generator network, and so a recall metric is provided to evaluate how well the generator captures the space of plausible images.

Compressed Machine Learning-Based Inverse Model for Design Optimization of Microwave Components

This article presents a new noniterative inverse modeling technique based on machine learning regression and its applications to microwave design optimization. The proposed inverse model accepts the high-dimensional S-parameters computed at many frequency points as the input and estimates the optimal geometrical/physical parameters of the microwave component as its output. The least-squares support vector machine regression is combined with the principal component analysis to simultaneously overcome both the high-dimensional input space and ill-posed challenges of the inverse modeling. We also propose a new empirical method to find the optimum number of principal components (i.e., the compression level) for each example in an automated way. This makes our proposed model general and easy to use compared with the existing data-driven inverse modeling techniques. The inverse model is trained by a set of scattering parameters computed via a 2-D/3-D solver for few configurations of the geometrical parameters. The feasibility and the accuracy of the proposed optimization scheme are investigated by comparing its predictions with the corresponding optimal configuration estimated via a commercial solver.