High-dimensional Input Research Articles

Dimensionality reduction for surrogate model construction for global sensitivity analysis: Comparison between active subspace and local sensitivity analysis

Global sensitivity analysis of combustion kinetic models is time consuming, because numerous samples generated by kinetic model simulations within the uncertainty space of the input parameters are needed. To alleviate this problem, surrogate models are often adopted to replace the kinetic model to generate samples. Directly constructing surrogate models is difficult in high-dimensional systems, therefore, local sensitivity analysis is usually first performed to exclude unimportant parameters. However, local sensitivity analysis may cause inaccurate screening when the model is nonlinear and the uncertainties of the input parameters are large. To overcome this difficulty, we propose herein the active subspace-based surrogate model (ASSM) method, which uses the active subspace to transform the high-dimensional input to low-dimensional features and then maps these features to the model target. The entire parameters are included in the ASSM while the mapping relationship is fitted with low-dimensional features. The accuracy and the computational cost of the ASSM are compared with those of the local sensitivity-based surrogate model (LSSM) in the hydrogen and methanol combustion systems. The results show that the ASSM can improve the data utilization and lead to a lower computational cost compared to the LSSM when several targets are considered, because one kinetic model simulation can yield original samples for many model targets simultaneously for the ASSM. The global sensitivity analysis with the ASSM and the LSSM is compared in the hydrogen, methanol and n-decane combustion systems, and the results show that the ASSM could lead to more reliable sensitivity indices than the LSSM because the active subspace-based dimensionality reduction can prevent the possibility of missing important parameters by transforming the entire parameters to new features instead of excluding some parameters.

Integrated deep learning neural network and desirability analysis in biogas plants: A powerful tool to optimize biogas purification

Performing anaerobic digestion is affected by different slurry properties, and its optimization poses many practical constraints. In high-dimensional input parameters with small sample size data, regression models and simple artificial neural network models may not be good enough at estimating responses. Therefore, a deep learning neural network (DNN) model was developed to estimate the responses (biogas compounds) according to the slurry properties. This model was able to predict the biogas compounds with high accuracy in comparison with regression models and back propagation neural network models. The DNN model was integrated with desirability analysis to determine optimum amounts of the slurry properties, and thus, increase biogas purification. Accordingly, the optimum ranges of C/N (15.04–18.95), BOD/COD (0.763–0.818), TS (8.1–10.6%) and T.VS (38.19–49.46%) were more precise than the ranges reported in the literature. The results indicated that large amounts of BOD/COD had a deterrent effect on desirability values, and therefore had an inhibitory effect on biogas purification. Further, pH amounts slightly above neutral could improve biogas purification. Suitable amounts of the slurry properties for the second repetition of experiments were all in the determined optimum ranges, indicating that the optimum ranges were practical to be used in biogas plants.

SWFC-ART: A cost-effective approach for Fixed-Size-Candidate-Set Adaptive Random Testing through small world graphs

Adaptive random testing (ART) improves the failure-detection effectiveness of random testing by leveraging properties of the clustering of failure-causing inputs of most faulty programs: ART uses a sampling mechanism that evenly spreads test cases within a software’s input domain. The widely-used Fixed-Sized-Candidate-Set ART (FSCS-ART) sampling strategy faces a quadratic time cost, which worsens as the dimensionality of the software input domain increases. In this paper, we propose an approach based on small world graphs that can enhance the computational efficiency of FSCS-ART: SWFC-ART. To efficiently perform nearest neighbor queries for candidate test cases, SWFC-ART incrementally constructs a hierarchical navigable small world graph for previously executed, non-failure-causing test cases. Moreover, SWFC-ART has shown consistency in programs with high dimensional input domains. Our simulation and empirical studies show that SWFC-ART reduces the computational overhead of FSCS-ART from quadratic to log-linear order while maintaining the failure-detection effectiveness of FSCS-ART, and remaining consistent in high dimensional input domains. We recommend using SWFC-ART in practical software testing scenarios, where real-life programs often have high dimensional input domains and low failure rates.

Fast Feature Selection for Linear Value Function Approximation

Linear value function approximation is a standard approach to solving reinforcement learning problems with large state spaces. Since designing good approximation features is difficult, automatic feature selection is an important research topic. We propose a new method for feature selection that is based on a low-rank factorization of the transition matrix. Our approach derives features directly from high-dimensional raw inputs, such as image data. The method is easy to implement using SVD, and our experiments show that it is faster and more stable than alternative methods.

High Dimensional Level Set Estimation with Bayesian Neural Network

Level Set Estimation (LSE) is an important problem with applications in various fields such as material design, biotechnology, machine operational testing, etc. Existing techniques suffer from the scalability issue, that is, these methods do not work well with high dimensional inputs. This paper proposes novel methods to solve the high dimensional LSE problems using Bayesian Neural Networks. In particular, we consider two types of LSE problems: (1) explicit LSE problem where the threshold level is a fixed user-specified value, and, (2) implicit LSE problem where the threshold level is defined as a percentage of the (unknown) maximum of the objective function. For each problem, we derive the corresponding theoretic information based acquisition function to sample the data points so as to maximally increase the level set accuracy. Furthermore, we also analyse theoretical time complexity of our proposed acquisition functions, and suggest a practical methodology to efficiently tune the network hyper-parameters to achieve high model accuracy. Numerical experiments on both synthetic and real-world datasets show that our proposed methods can achieve better results compared to existing state-of-the-art approaches.

Fuzzy-based active learning for predicting student academic performance using autoML: a step-wise approach

Predicting students’ learning outcomes is one of the main topics of interest in the area of Educational Data Mining and Learning Analytics. To this end, a plethora of machine learning methods has been successfully applied for solving a variety of predictive problems. However, it is of utmost importance for both educators and data scientists to develop accurate learning models at low cost. Fuzzy logic constitutes an appropriate approach for building models of high performance and transparency. In addition, active learning reduces both the time and cost of labeling effort, by exploiting a small set of labeled data along with a large set of unlabeled data in the most efficient way. In addition, choosing the proper method for a given problem formulation and configuring the optimal parameter setting is a demanding task, considering the high-dimensional input space and the complexity of machine learning algorithms. As such, exploring the potential of automated machine learning (autoML) strategies from the perspective of machine learning adeptness is important. In this context, the present study introduces a fuzzy-based active learning method for predicting students’ academic performance which combines, in a modular way, autoML practices. A lot of experiments was carried out, revealing the efficiency of the proposed method for the accurate prediction of students at risk of failure. These insights may have the potential to support the learning experience and be useful the wider science of learning.

Deep Neural Network Behavioral Modeling Based on Transfer Learning for Broadband Wireless Power Amplifier

The behavior model based on the artificial neural network has been widely used in the broadband power amplifier (PA). Although the deep neural network (DNN) performs well in the PA modeling with high-dimensional inputs, the training time of the DNN model is still long. This letter proposes a PA modeling method based on transfer learning to reduce training time without sacrificing modeling performance. In the proposed method, the model can be divided into two parts. The first part is defined as a predesigned filter that can extract the features of PA, and the second part is defined as adaptation layers that can be used to fit the real PA output. Experimental results show that the proposed method can effectively reduce the training time and ensure good modeling performance compared with the traditional DNN model.

Assurance monitoring of learning-enabled cyber-physical systems using inductive conformal prediction based on distance learning

AbstractMachine learning components such as deep neural networks are used extensively in cyber-physical systems (CPS). However, such components may introduce new types of hazards that can have disastrous consequences and need to be addressed for engineering trustworthy systems. Although deep neural networks offer advanced capabilities, they must be complemented by engineering methods and practices that allow effective integration in CPS. In this paper, we proposed an approach for assurance monitoring of learning-enabled CPS based on the conformal prediction framework. In order to allow real-time assurance monitoring, the approach employs distance learning to transform high-dimensional inputs into lower size embedding representations. By leveraging conformal prediction, the approach provides well-calibrated confidence and ensures a bounded small error rate while limiting the number of inputs for which an accurate prediction cannot be made. We demonstrate the approach using three datasets of mobile robot following a wall, speaker recognition, and traffic sign recognition. The experimental results demonstrate that the error rates are well-calibrated while the number of alarms is very small. Furthermore, the method is computationally efficient and allows real-time assurance monitoring of CPS.

Imputation of missing data from offshore wind farms using spatio-temporal correlation and feature correlation

Many industrial applications, such as fault diagnosis and remaining useful life prediction, require high-dimensional inputs to predict a reliable output. For offshore wind farm supervisory control and data acquisition (SCADA) systems, unfortunately, signal inputs are often missing due to harsh weather, resulting in the failure of network/sensors. It limits the accuracy of subsequent diagnostic or prognostic tasks. Although many methods have been proposed for imputing missing data, their applicability in offshore wind farms is still problematic because wind turbines (WTs) are time-varying systems, and conventional learning methods require high computational cost. To address this problem, we propose a learning framework containing two learning models, corresponding to two missing-data conditions. The framework imputes missing data by designing a spatio-temporal correlation method for entire feature-missing conditions and a feature-correlation method for partial feature-missing conditions, respectively. A real-world offshore wind farm dataset of a SCADA system with 33 WTs and 68 features, which was recorded over a one-month period, is used for experimental validation. We demonstrate that the proposed framework imputes the missing data with much smaller mean absolute error (MAE) and mean squared error (MSE) and requires less computational time, compared to the existing machine-learning methods for both imputation conditions.

An Efficient Communication Intrusion Detection Scheme in AMI Combining Feature Dimensionality Reduction and Improved LSTM

Communication intrusion detection in Advanced Metering Infrastructure (AMI) is an eminent security technology to ensure the stable operation of the Smart Grid. However, methods based on traditional machine learning are not appropriate for learning high-dimensional features and dealing with the data imbalance of communication traffic in AMI. To solve the above problems, we propose an intrusion detection scheme by combining feature dimensionality reduction and improved Long Short-Term Memory (LSTM). The Stacked Autoencoder (SAE) has shown excellent performance in feature dimensionality reduction. We compress high-dimensional feature input into low-dimensional feature output through SAE, narrowing the complexity of the model. Methods based on LSTM have a superior ability to detect abnormal traffic but cannot extract bidirectional structural features. We designed a Bi-directional Long Short-Term Memory (BiLSTM) model that added an Attention Mechanism. It can determine the criticality of the dimensionality and improve the accuracy of the classification model. Finally, we conduct experiments on the UNSW-NB15 dataset and the NSL-KDD dataset. The proposed scheme has obvious advantages in performance metrics such as accuracy and False Alarm Rate (FAR). The experimental results demonstrate that it can effectively identify the intrusion attack of communication in AMI.

Efficient search over incomplete knowledge graphs in binarized embedding space

Knowledge graph (KG) embedding techniques represent entities and relations as low-dimensional and continuous vectors. This enables KG machine learning models to be easily adapted for KG reasoning, completion, and querying tasks. However, learned dense vectors are inefficient for large-scale similarity computations. Learning-to-hash is to a method that learns compact binary codes from high-dimensional input data and provides a promising way to accelerate efficiency by measuring the Hamming distance instead of Euclidean distance. Alternatively, a dot-product is used in a continuous vector space. Unfortunately, most learning-to-hash methods cannot be directly applied to KG structure encoding because they focus on similarity preservation between images. In this paper, we introduce a novel end-to-end learning-to-hash framework for encoding incomplete KGs and graph queries in a Hamming space. To preserve KG structure information, from embeddings to hash codes, and address the ill-posed gradient issue in the optimization, we utilize a continuation method (with convergence guarantees) to jointly encode queries and KG entities using geometric operations. The hashed embedding of a query can be utilized to discover target entities from incomplete KGs whilst the efficiency has been greatly improved. To evaluate the proposed framework, we have compared our model to state-of-the-art methods commonly used in real-world KGs. Extensive experimental results show that our framework not only significantly speeds up the search process, but also provides good results when unanswerable queries are caused by incomplete information.11Our code and data are open in Github https://github.com/seu-kse/HashKG.

Large scale multi-label learning using Gaussian processes

We introduce a Gaussian process latent factor model for multi-label classification that can capture correlations among class labels by using a small set of latent Gaussian process functions. To address computational challenges, when the number of training instances is very large, we introduce several techniques based on variational sparse Gaussian process approximations and stochastic optimization. Specifically, we apply doubly stochastic variational inference that sub-samples data instances and classes which allows us to cope with Big Data. Furthermore, we show it is possible and beneficial to optimize over inducing points, using gradient-based methods, even in very high dimensional input spaces involving up to hundreds of thousands of dimensions. We demonstrate the usefulness of our approach on several real-world large-scale multi-label learning problems.

A dimension-reduction metamodeling approach to simulation-based uncertainty quantification problems with high dimensionalities

A common strategy to handle simulation-based uncertainty quantification problems is adopting a metamodel to replace time-demanding calculations such as computational fluid dynamics simulation or finite element analysis within Monte Carlo simulation process. However, most of the so far metamodel-assisted uncertainty quantification methods suffer from the ‘curse of dimensionality.’ The required number of evaluations, which determines the computational cost, increases exponentially as the dimensionality of the input uncertainty increases, resulting in unaffordable computational cost for high-dimensional problems. Another challenge emerges when the output uncertainties are a spatially varying field accommodating a huge number of spatial nodes. To solve these issues, here we propose a dimension-reduction metamodeling approach, in which active subspace method is utilized to reduce the input dimensionality and proper orthogonal decomposition method is utilized to reduce the output dimensionality of the spatially varying field. The relationship between the two methods is established by using the support vector regression model. Through uncertainty quantification of seven stochastic analytical functions and one stochastic convection-diffusion equation, the proposed approach was verified to be fairly accurate in propagating high-dimensional input uncertainties to either a scalar value or a spatially varying output. The accuracy and efficiency of the proposed approach in dealing with even more practical simulation-based problems were then validated by uncertainty quantification of a compressor cascade with stochastic protrusions/dents distributed on the blade surface. This work provides an effective and versatile approach for simulation-based high-dimensional uncertainty quantification problems.

Action Sequencing Using Visual Permutations

Humans can easily reason about the sequence of high level actions needed to complete tasks, but it is particularly difficult to instill this ability in robots trained from relatively few examples. This work considers the task of neural action sequencing conditioned on a single reference visual state. This task is extremely challenging as it is not only subject to the significant combinatorial complexity that arises from large action sets, but also requires a model that can perform some form of symbol grounding, mapping high dimensional input data to actions, while reasoning about action relationships. This letter takes a permutation perspective and argues that action sequencing benefits from the ability to reason about both permutations and ordering concepts. Empirical analysis shows that neural models trained with latent permutations outperform standard neural architectures in constrained action sequencing tasks. Results also show that action sequencing using visual permutations is an effective mechanism to initialise and speed up traditional planning techniques and successfully scales to far greater action set sizes than models considered previously.

LDA-Reg: Knowledge Driven Regularization Using External Corpora

While recent developments of neural network (NN) models have led to a series of record-breaking achievements in many applications, the lack of sufficiently good datasets remains a problem for some applications. For such a problem, we can however exploit a large number of unstructured text corpora as an external knowledge to complement the training data, and most prevailing neural network solutions employ word embedding methods for such purposes. In this paper, we propose LDA-Reg, a novel knowledge driven regularization framework based on Latent Dirichlet Allocation (LDA) as an alternative to the word embedding methods to adaptively utilize abundant external knowledge and to interpret the NN model. For the joint learning of the parameters, we propose EM-SGD, an effective update method which incorporates Expectation Maximization (EM) and Stochastic Gradient Descent (SGD) to update parameters iteratively. Moreover, we also devise a lazy update and sparse update method for the high-dimensional inputs and sparse inputs respectively. We validate the effectiveness of our regularization framework through an extensive experimental study over real world and standard benchmark datasets. The results show that our proposed framework not only achieves significant improvement over state-of-the-art word embedding methods but also learns interpretable and significant topics for various tasks.

The importance of uncertainty quantification in model reproducibility.

Many computer models possess high-dimensional input spaces and substantial computational time to produce a single model evaluation. Although such models are often ‘deterministic’, these models suffer from a wide range of uncertainties. We argue that uncertainty quantification is crucial for computer model validation and reproducibility. We present a statistical framework, termed history matching, for performing global parameter search by comparing model output to the observed data. We employ Gaussian process (GP) emulators to produce fast predictions about model behaviour at the arbitrary input parameter settings allowing output uncertainty distributions to be calculated. History matching identifies sets of input parameters that give rise to acceptable matches between observed data and model output given our representation of uncertainties. Modellers could proceed by simulating computer models’ outputs of interest at these identified parameter settings and producing a range of predictions. The variability in model results is crucial for inter-model comparison as well as model development. We illustrate the performance of emulation and history matching on a simple one-dimensional toy model and in application to a climate model.This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico’.

Short-Term Wind Power Prediction for Wind Farm Clusters Based on SFFS Feature Selection and BLSTM Deep Learning

Wind power prediction (WPP) of wind farm clusters is important to the safe operation and economic dispatch of the power system, but it faces two challenges: (1) The dimensions of the input parameters for WPP of wind farm clusters are very high so that the input parameters contain irrelevant or redundant features; (2) it is difficult to build a holistic WPP model with high-dimensional input parameters for wind farm clusters. To overcome these challenges, a novel short-term WPP model for wind farm clusters, based on sequential floating forward selection (SFFS) feature selection and bidirectional long short-term memory (BLSTM) deep learning, is proposed in this paper. First, more than 300,000 input features of the wind farm cluster are constructed. Second, the SFFS method is applied to sort the high-dimensional features and analyze the rule that the forecasting accuracy changes with the number of features to obtain the optimal number of features and feature sets. Finally, based on the results of feature selection, BLSTM is applied to build a WPP model for wind farm clusters with a combination of feature selection and deep learning. This case study shows that (1) SFFS is an effective method for selecting the core features for WPP of wind farm clusters; (2) BLSTM shows not only higher WPP accuracy than long short-term memory and backpropagation neural network but also outstanding performance in terms of reducing the phase errors of WPP.

Scalable Fully Bayesian Gaussian Process Modeling and Calibration With Adaptive Sequential Monte Carlo for Industrial Applications

Abstract Gaussian process (GP) regression or kriging has been extensively applied in the engineering literature for the purposes of building a cheap-to-evaluate surrogate, within the contexts of multi-fidelity modeling, model calibration, and design optimization. With the ongoing automation of manufacturing and industrial practices as a part of Industry 4.0, there has been a greater need for advancing GP regression techniques to handle challenges such as high input dimensionality, data paucity or big data problems, these consist primarily of proposing efficient design of experiments, optimal data acquisition strategies, sparsifying covariance kernels, and other mathematical tricks. In this work, our attention is focused on the challenges of efficiently training a GP model, which, to the authors opinion, has attracted very little attention and is to-date poorly addressed. The performance of widely used training approaches such as maximum likelihood estimation and Markov Chain Monte Carlo (MCMC) sampling can deteriorate significantly in high-dimensional and big data problems and can lead to cost deficient implementations of critical importance to many industrial applications. Here, we compare an Adaptive Sequential Monte Carlo (ASMC) sampling algorithm to classic MCMC sampling strategies and we demonstrate the effectiveness of our implementation on several mathematical problems and challenging industry applications of varying complexity. The computational time savings of the ASMC approach manifest in large-scale problems helping us to push the boundaries of applicability and scalability of GPs for model calibration in various domains of the industry, including but not limited to design automation, design engineering, smart manufacturing, predictive maintenance, and supply chain manufacturing.

Model-Based Design of Experiments for High-Dimensional Inputs Supported by Machine-Learning Methods

Algorithms that compute locally optimal continuous designs often rely on a finite design space or on the repeated solution of difficult non-linear programs. Both approaches require extensive evaluations of the Jacobian Df of the underlying model. These evaluations are a heavy computational burden. Based on the Kiefer-Wolfowitz Equivalence Theorem, we present a novel design of experiments algorithm that computes optimal designs in a continuous design space. For this iterative algorithm, we combine an adaptive Bayes-like sampling scheme with Gaussian process regression to approximate the directional derivative of the design criterion. The approximation allows us to adaptively select new design points on which to evaluate the model. The adaptive selection of the algorithm requires significantly less evaluations of Df and reduces the runtime of the computations. We show the viability of the new algorithm on two examples from chemical engineering.

Mapping machine-learned physics into a human-readable space

We present a technique for translating a black-box machine-learned classifier operating on a high-dimensional input space into a small set of human-interpretable observables that can be combined to make the same classification decisions. We iteratively select these observables from a large space of high-level discriminants by finding those with the highest decision similarity relative to the black box, quantified via a metric we introduce that evaluates the relative ordering of pairs of inputs. Successive iterations focus only on the subset of input pairs that are misordered by the current set of observables. This method enables simplification of the machine-learning strategy, interpretation of the results in terms of well-understood physical concepts, validation of the physical model, and the potential for new insights into the nature of the problem itself. As a demonstration, we apply our approach to the benchmark task of jet classification in collider physics, where a convolutional neural network acting on calorimeter jet images outperforms a set of six well-known jet substructure observables. Our method maps the convolutional neural network into a set of observables called energy flow polynomials, and it closes the performance gap by identifying a class of observables with an interesting physical interpretation that has been previously overlooked in the jet substructure literature.