High-dimensional Input Research Articles

RockGPT: reconstructing three-dimensional digital rocks from single two-dimensional slice with deep learning

Random reconstruction of three-dimensional (3D) digital rocks from two-dimensional (2D) slices is crucial for elucidating the microstructure of rocks and its effects on pore-scale flow in terms of numerical modeling, since massive samples are usually required to handle intrinsic uncertainties. Despite remarkable advances achieved by traditional process-based methods, statistical approaches and recently famous deep learning-based models, few works have focused on producing several kinds of rocks with one trained model and allowing the reconstructed samples to approximately satisfy certain given properties, such as porosity. To fill this gap, we propose a new framework with deep learning, named RockGPT, which is composed of VQ-VAE and conditional GPT, to synthesize 3D samples based on a single 2D slice from the perspective of video generation. The VQ-VAE is utilized to compress high-dimensional input video, i.e., the sequence of continuous rock slices, to discrete latent codes and reconstruct them. In order to obtain diverse reconstructions, the discrete latent codes are modeled using conditional GPT in an autoregressive manner, while incorporating conditional information from a given slice, rock type, and porosity. We conduct two experiments on five kinds of rocks, and the results demonstrate that RockGPT can produce different kinds of rocks with a single model, and the porosities of reconstructed samples can distribute around specified targets with a narrow range. In a broader sense, through leveraging the proposed conditioning scheme, RockGPT constitutes an effective way to build a general model to produce multiple kinds of rocks simultaneously that also satisfy user-defined properties.

Data-driven models for train control dynamics in high-speed railways: LAG-LSTM for train trajectory prediction

The construction of an accurate train control model (TCM) is crucial to the design of automatic train operation and real-time traffic management systems in high-speed railways. Traditional physical-driven models usually fail to reflect the “true” dynamics of high-speed trains (HSTs) because of the strong nonlinearity and uncertainty due to air resistance, frequently switching working conditions, and variations in external influencing factors such as weather and temperature. Although some data-driven deep learning models have recently been proposed for environmental adaptation, they are all “black-box” models, which cannot explain how the input of the models affects the HST output. To overcome these issues, this study constructs a novel long short-term memory with lagged information (LAG-LSTM) model by combining the physical-driven HST model and an “interpretable” deep learning model. Specifically, our LAG-LSTM model contains three modules: a time-delay variable module to model the transform delay of control variables, state variable enhancement module to extract the key features among high-dimensional input data, and Pre-LSTM module to predict the future train trajectory with given control variables. We collected field data from the Beijing-Shanghai high-speed railway and developed a data filter method and a normalization procedure to overcome the positioning errors of HSTs and construct a standard data set. Finally, we tested the effectiveness of our LAG-LSTM by comparing it with six deep learning structures, including fully connected neural network, recursive neural network, standard LSTM, and LSTM with convolutional layers. The results show that LAG-LSTM can accurately predict the trajectories of HSTs and outperforms other deep learning models. Regarding prediction accuracy, LAG-LSTM improved the performance of the traditional LSTM by 13.5% to 23.3%.

On High-Dimensional Time-Variant Reliability Analysis with the Maximum Entropy Principle

The structural reliability analysis suffers from the curse of dimensionality if the associated limit state function involves a large number of inputs. This study develops a reliability analysis method that deals with high-dimensional inputs over time. The probability distribution of the structural response is reconstructed by the maximum entropy principle which is achieved by solving an optimization problem derived from the concept of relative entropy. The optimization problem is transformed into a convex one with respect to the orders of fractional moments and the Lagrange multipliers. Additionally, considering the associated computational issues, it is reformulated with side constraints on the parameters of the maximum entropy distribution. Then, a global optimization procedure is performed. The proposed method is successfully applied to the reliability analysis of a linear and a nonlinear structural system, which involves a large number of inputs deriving from the discretization of the input random processes.

Enhanced SCADA IDS Security by Using MSOM Hybrid Unsupervised Algorithm

In Self-Organizing Maps (SOM) are unsupervised neural networks that cluster high dimensional data and transform complex inputs into easily understandable inputs. To find the closest distance and weight factor, it maps high dimensional input space to low dimensional input space. The Closest node to data point is denoted as a neuron. It classifies the input data based on these neurons. The reduction of dimensionality and grid clustering using neurons makes to observe similarities between the data. In our proposed Mutated Self Organizing Maps (MSOM) approach, we have two intentions. One is to eliminate the learning rate and to decrease the neighborhood size and the next one is to find out the outliers in the network. The first one is by calculating the median distance (MD) between each node with its neighbor nodes. Then those median values are compared with one another. In case, if any of the MD values significantly varies from the rest then it is declared as anomaly nodes. In the second phase, we find out the quantization error (QE) in each instance from the cluster center.

Classification of steel using laser-induced breakdown spectroscopy combined with deep belief network.

The identification of steels is a crucial step in the process of recycling and reusing steel waste. Laser-induced breakdown spectroscopy (LIBS) coupled with machine learning is a convenient method to classify the types of materials. LIBS can generate characteristic spectra of various samples as input variable for steel classification in real time. However, the performance of classification model is limited to the complex input due to similar chemical composition in samples and nonlinearity problems between spectral intensities and elemental concentrations. In this study, we developed a method of LIBS coupled with deep belief network (DBN), which is suitable to deal with a nonlinear problem, to classify 13 brands of special steels. The performance of the training and validation sets were used as the standard to optimize the structure of DBN. For different input, such as the intensities of full-spectra signals and characteristic spectra lines, the accuracies of the optimized DBN model in the training, validation, and test set are all over 98%. Moreover, compared with the self-organizing maps, linear discriminant analysis (LDA), k-nearest neighbor (KNN) and back-propagation artificial neural networks (BPANN), the result of the test set showed that the optimized DBN model performed second best (98.46%) in all methods using characteristic spectra lines as input. The test accuracy of the DBN model could reach 100% and the maximum accuracy of other methods ranged from 62.31% to 96.16% using full-spectra signals as input. This study demonstrates that DBN can extract representative feature information from high-dimensional input, and that LIBS coupled with DBN has great potential for steel classification.

Structural damage identification method based on vibration statistical indicators and support vector machine

This study presents a support vector machine (SVM) based structural damage detection approach for a steel frame model. Totally 10 structural scenarios including one undamaged and nine damaged cases (D1–D9) are defined by replacing different column elements to reduce the lateral stiffness between layers. Sensors are laid out at each layer of the frame structure to pick up acceleration response stirred up by the simulated white noise excitations. Statistical analysis is carried out and three damage indictor features are extracted from 800 samples of record duration. Subsequently, the SVM classifiers are developed to conduct a series of indictor fusion schemes to identify multiple damage states. The results indicate that the approach has good identification performance with damage features from three indicators, especially for a high dimensional input eigenvector. Meanwhile, acceptable detection accuracy can be obtained in situations with incomplete measurement; the robustness in terms of the resistance against noise is also discussed.

PQROM: To optimize software defined network QoS-aware routing with proximal policy optimization

Currently, Quality-of-Service (QoS)-aware routing is one of the crucial challenges in Software Defined Network (SDN). The QoS performances, e.g. latency, packet loss ratio and throughput, must be optimized to improve the performance of network. Traditional static routing algorithms based on Open Shortest Path First (OSPF) could not adapt to traffic fluctuation, which may cause severe network congestion and service degradation. Central intelligence of SDN controller and recent breakthroughs of Deep Reinforcement Learning (DRL) pose a promising solution to tackle this challenge. Thus, we propose an on-policy DRL mechanism, namely the PPO-based (Proximal Policy Optimization) QoS-aware Routing Optimization Mechanism (PQROM), to achieve a general and re-customizable routing optimization. PQROM can dynamically update the routing calculation by adjusting the reward function according to different optimization objectives, and it is independent of any specific network pattern. Additionally, as a black-box one-step optimization, PQROM is qualified for both continuous and discrete action space with high-dimensional input and output. The OMNeT ++ simulation experiment results show that PQROM not only has good convergence, but also has better stability compared with OSPF, less training time and simpler hyper-parameters adjustment than Deep Deterministic Policy Gradient (DDPG) and less hardware consumption than Asynchronous Advantage Actor-Critic (A3C).

ENHANCED SCADA IDS SECURITY BY USING MSOM HYBRID UNSUPERVISED ALGORITHM

In Self-Organizing Maps (SOM) are unsupervised neural networks that cluster high dimensional data and transform complex inputs into easily understandable inputs. To find the closest distance and weight factor, it maps high dimensional input space to low dimensional input space. The Closest node to data point is denoted as a neuron. It classifies the input data based on these neurons. The reduction of dimensionality and grid clustering using neurons makes to observe similarities between the data. In our proposed Mutated Self Organizing Maps (MSOM) approach, we have two intentions. One is to eliminate the learning rate and to decrease the neighborhood size and the next one is to find out the outliers in the network. The first one is by calculating the median distance (MD) between each node with its neighbor nodes. Then those median values are compared with one another. In case, if any of the MD values significantly varies from the rest then it is declared as anomaly nodes. In the second phase, we find out the quantization error (QE) in each instance from the cluster center.

Varying Coefficient Models and Design Choice for Bayes Linear Emulation of Complex Computer Models with Limited Model Evaluations

Computer models are widely used to help make decisions about real-world systems. As computer models of large and complex systems can have long run-times and high-dimensional input spaces, it is often necessary to use emulation to assess uncertainties in computer model output. This paper presents methodology for emulation of complex computer models motivated by a real-world example in energy policy. The computer model studied is an economic model of investment in electricity generation in Great Britain. The computer model was used to select parameters in a government policy designed to incentivize investment in renewable technologies to meet government targets. Limited computing time meant that few runs of the computer model were available to fit an emulator. The statistical methodology developed was therefore focused on accurately capturing the uncertainty in computer model output arising from the small number of available model runs. A varying coefficient emulator is proposed to model uncertainty in model output when extrapolating away from model runs. To maximize use of the small number of runs available, this varying coefficient emulator is paired with a criterion-based procedure for design selection.

Accurate Remaining Useful Life Prediction With Uncertainty Quantification: A Deep Learning and Nonstationary Gaussian Process Approach

Remaining useful life (RUL) refers to the expected remaining lifespan of a component or system. Accurate RUL prediction is critical for prognostic and health management and for maintenance planning. In this article, we address three prevalent challenges in data-driven RUL prediction, namely the handling of high-dimensional input features, the robustness to noise in sensor data and prognostic datasets, and the capturing of the time-dependency between system degradation and RUL prediction. We devise a highly accurate RUL prediction model with uncertainty quantification, which integrates and leverages the advantages of deep learning and nonstationary Gaussian process regression (DL–NSGPR). We examine and benchmark our model against other advanced data-driven RUL prediction models using the turbofan engine dataset from the NASA prognostic repository. Our computational experiments show that the DL-NSGPR predictions are highly accurate with root mean square error 1.7 to 6.2 times smaller than those of competing RUL models. Furthermore, the results demonstrate that RUL uncertainty bounds with the proposed DL-NSGPR are both valid and significantly tighter than other stochastic RUL prediction models. We unpack and discuss the reasons for this excellent performance of the DL-NSGPR.

Sparse-Coding-Based Autoencoder and Its Application for Cancer Survivability Prediction

Cancer-survivability prediction is one of the popular research topics, that attracted great attention from both the health service providers and academia. However, one remaining question comes from how to make full use of a large number of available factors (or features). This paper, accordingly, presents a novel autoencoder algorithm based on the concept of sparse coding to address this problem. The main contribution is twofold: the utilization of sparsity coding for input feature selection and a subsequent classification using latent information. Precisely, a typical autoencoder architecture is employed for reconstructing the original input. Then the sparse coding technique is applied to optimize the network structure, with the aim of selecting optimal features and enhancing the generalization capability. In addition, the refined latent information is further cast as alternative features for training a sparse classifier. To evaluate the performance of the proposed autoencoder architecture, we present a comprehensive analysis using a publicly available data repository (i.e., Surveillance, Epidemiology, and End Results, SEER). Experimental study shows that the proposed approach has the ability of extracting important features from high-dimensional inputs and achieves competitive performance than other state-of-the-art classification techniques.

A weight initialization method based on neural network with asymmetric activation function

Weight initialization of neural networks has an important influence on the learning process, and the selection of initial weights is related to the activation interval of the activation function. It is proposed that an improved and extended weight initialization method for neural network with asymmetric activation function as an extension of the linear interval tolerance method (LIT), called ‘GLIT’ (generalized LIT), which is more suitable for higher-dimensional inputs. The purpose is to expand the selection range of the activation function so that the input falls in the unsaturated region, so as to improve the performance of the network. Then, a tolerance solution theorem based upon neural network system is given and proved. Furthermore, the algorithm is given about determining the initial weight interval. The validity of the theorem and algorithm is verified by numerical experiments. The input could fall into any preset interval in the sense of probability under the GLIT method. In another sense, the GLIT method could provide a theoretical basis for the further study of neural networks.

Explaining adversarial vulnerability with a data sparsity hypothesis

Despite many proposed algorithms to provide robustness to deep learning (DL) models, DL models remain susceptible to adversarial attacks. We hypothesize that the adversarial vulnerability of DL models stems from two factors. The first factor is data sparsity which is that in the high dimensional input data space, there exist large regions outside the support of the data distribution. The second factor is the existence of many redundant parameters in the DL models. Owing to these factors, different models are able to come up with different decision boundaries with comparably high prediction accuracy. The appearance of the decision boundaries in the space outside the support of the data distribution does not affect the prediction accuracy of the model. However, it makes an important difference in the adversarial robustness of the model. We hypothesize that the ideal decision boundary is as far as possible from the support of the data distribution. In this paper, we develop a training framework to observe if DL models are able to learn such a decision boundary spanning the space around the class distributions further from the data points themselves. Semi-supervised learning was deployed during training by leveraging unlabeled data generated in the space outside the support of the data distribution. We measured adversarial robustness of the models trained using this training framework against well-known adversarial attacks and by using robustness metrics. We found that models trained using our framework, as well as other regularization methods and adversarial training support our hypothesis of data sparsity and that models trained with these methods learn to have decision boundaries more similar to the aforementioned ideal decision boundary. We show that the unlabeled data generated by noise in our framework is almost as effective on adversarial robustness as unlabeled data sourced from existing datasets or generated by synthesis algorithms. The code for our training framework is available online.

Using BART to Perform Pareto Optimization and Quantify its Uncertainties

Techniques to reduce the energy burden of an industrial ecosystem often require solving a multiobjective optimization problem. However, collecting experimental data can often be either expensive or time-consuming. In such cases, statistical methods can be helpful. This article proposes Pareto Front (PF) and Pareto Set (PS) estimation methods using Bayesian Additive Regression Trees (BART), which is a nonparametric model whose assumptions are typically less restrictive than popular alternatives, such as Gaussian Processes (GPs). These less restrictive assumptions allow BART to handle scenarios (e.g., high-dimensional input spaces, nonsmooth responses, large datasets) that GPs find difficult. The performance of our BART-based method is compared to a GP-based method using analytic test functions, demonstrating convincing advantages. Finally, our BART-based methodology is applied to a motivating engineering problem. Supplementary materials, which include a theorem proof, algorithms, and R code, for this article are available online.

Machine learning classification of trajectories from molecular dynamics simulations of chromosome segregation.

In contrast to the well characterized mitotic machinery in eukaryotes it seems as if there is no universal mechanism organizing chromosome segregation in all bacteria. Apparently, some bacteria even use combinations of different segregation mechanisms such as protein machines or rely on physical forces. The identification of the relevant mechanisms is a difficult task. Here, we introduce a new machine learning approach to this problem. It is based on the analysis of trajectories of individual loci in the course of chromosomal segregation obtained by fluorescence microscopy. While machine learning approaches have already been applied successfully to trajectory classification in other areas, so far it has not been possible to use them to discriminate segregation mechanisms in bacteria. A main obstacle for this is the large number of trajectories required to train machine learning algorithms that we overcome here by using trajectories obtained from molecular dynamics simulations. We used these trajectories to train four different machine learning algorithms, two linear models and two tree-based classifiers, to discriminate segregation mechanisms and possible combinations of them. The classification was performed once using the complete trajectories as high-dimensional input vectors as well as on a set of features which were used to transform the trajectories into low-dimensional input vectors for the classifiers. Finally, we tested our classifiers on shorter trajectories with duration times comparable (or even shorter) than typical experimental trajectories and on trajectories measured with varying temporal resolutions. Our results demonstrate that machine learning algorithms are indeed capable of discriminating different segregation mechanisms in bacteria and to even resolve combinations of the mechanisms on rather short time scales.

Research of Text Classification Based on TF-IDF and CNN-LSTM

With the rapid development of deep learning, many deep learning models have been widely used in Natural Language Processing(NLP). The Long-Short Term memory network(LSTM) model and convolutional neural network(CNN) model can achieve high classification accuracy in text classification tasks. However, the high input dimension of text features and the need to train a large number of parameters in the deep learning model often take a lot of time. This paper uses Term Frequency-inverse Document Frequency(TF-IDF) to remove features with lower weights, extract key features in the text, extract the corresponding word vector through the Word2Vec model, and then input it into the CNN-LSTM model. We compared the model with CNN, LSTM, and LSTM-attention methods and found that the model can significantly reduce model parameters and training time in short and long text data sets. The model hardly loses accuracy in the long text, but the model will lose a certain amount of accuracy in short texts. This paper also proposes fusing original text features to make up for the accuracy loss caused by the TF-IDF feature extraction method.

Algorithm and Hardware Co-Design of Energy-Efficient LSTM Networks for Video Recognition with Hierarchical Tucker Tensor Decomposition

Long short-term memory (LSTM) is a type of powerful deep neural network that has been widely used in many sequence analysis and modeling applications. However, the large model size problem of LSTM networks make their practical deployment still very challenging, especially for the video recognition tasks that require high-dimensional input data. Aiming to overcome this limitation and fully unlock the potentials of LSTM models, in this paper we propose to perform algorithm and hardware co-design towards high-performance energy-efficient LSTM networks. At algorithm level, we propose to develop <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fully decomposed hierarchical Tucker (FDHT)</i> structure-based LSTM, namely FDHT-LSTM, which enjoys ultra-low model complexity while still achieving high accuracy. In order to fully reap such attractive algorithmic benefit, we further develop the corresponding customized hardware architecture to support the efficient execution of the proposed FDHT-LSTM model. With the delicate design of memory access scheme, the complicated matrix transformation can be efficiently supported by the underlying hardware without any access conflict in an on-the-fly way. Our evaluation results show that both the proposed ultra-compact FDHT-LSTM models and the corresponding hardware accelerator achieve very high performance. Compared with the state-of-the-art compressed LSTM models, FDHT-LSTM enjoys both order-of-magnitude reduction (more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1000 \times$</tex-math></inline-formula> ) in model size and significant accuracy improvement (0.6% to 12.7%) across different video recognition datasets. Meanwhile, compared with the state-of-the-art tensor decomposed model-oriented hardware TIE, our proposed FDHT-LSTM architecture achieve <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2.5\times$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.46\times$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2.41\times$</tex-math></inline-formula> increase in throughput, area efficiency and energy efficiency, respectively on LSTM-Youtube workload. For LSTM-UCF workload, our proposed design also outperforms TIE with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.9\times$</tex-math></inline-formula> higher throughput, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.83\times$</tex-math></inline-formula> higher energy efficiency and comparable area efficiency.

Graph Embedding via High Dimensional Model Representation for Hyperspectral Images

Learning the manifold structure of remote sensing images is of paramount relevance for modeling and understanding processes, as well as encapsulating the high dimensionality in a reduced set of informative features for subsequent classification, regression, or unmixing. Manifold learning methods have shown excellent performance when dealing with hyperspectral image (HSI) analysis, but, unless specifically designed, they cannot provide an explicit embedding map readily applicable to out-of-sample (OOS) data. A common assumption to deal with the problem is that the transformation between the high-dimensional input space and the latent space (typically low) is linear. This is a particularly strong assumption, especially when dealing with HSIs due to the well-known nonlinear nature of the data. To address this problem, a manifold learning method based on high-dimensional model representation (HDMR) is proposed, which enables a nonlinear embedding function to project OOS samples into the latent space. The proposed method is compared to manifold learning methods along with their linear counterparts and achieves promising performance in terms of classification accuracy for a representative set of HSIs.

Physics-informed Neural Networks-based Model Predictive Control for Multi-link Manipulators

We discuss nonlinear model predictive control (MPC) for multi-body dynamics via physics-informed machine learning methods. In more detail, we use a physics-informed neural networks (PINNs)-based MPC to solve a tracking problem for a complex mechanical system, a multi-link manipulator. PINNs are a promising tool to approximate (partial) differential equations but are not suited for control tasks in their original form since they are not designed to handle variable control actions or variable initial values. We thus follow the strategy of Antonelo et al. (arXiv:2104.02556, 2021) by enhancing PINNs with adding control actions and initial conditions as additional network inputs. Subsequently, the high-dimensional input space is reduced via a sampling strategy and a zero-hold assumption. This strategy enables the controller design based on a PINN as an approximation of the underlying system dynamics. The additional benefit is that the sensitivities are easily computed via automatic differentiation, thus leading to efficient gradient-based algorithms for the underlying optimal control problem.

A FULLY BAYESIAN GRADIENT-FREE SUPERVISED DIMENSION REDUCTION METHOD USING GAUSSIAN PROCESSES

Modern day engineering problems are ubiquitously characterized by sophisticated computer codes that map parameters or inputs to an underlying physical process. In other situations, experimental setups are used to model the physical process in a laboratory, ensuring high precision while being costly in materials and logistics. In both scenarios, only limited amount of data can be generated by querying the expensive information source at a finite number of inputs or designs. This problem is compounded further in the presence of a high-dimensional input space. State-of-the-art parameter space dimension reduction methods, such as active subspace, aim to identify a subspace of the original input space that is sufficient to explain the output response. These methods are restricted by their reliance on gradient evaluations or copious data, making them inadequate to expensive problems without direct access to gradients. The proposed methodology is gradient-free and fully Bayesian, as it quantifies uncertainty in both the low-dimensional subspace and the surrogate model parameters. This enables a full quantification of epistemic uncertainty and robustness to limited data availability. It is validated on multiple datasets from engineering and science and compared to two other state-of-the-art methods based on four aspects: a) recovery of the active subspace, b) deterministic prediction accuracy, c) probabilistic prediction accuracy, and d) training time. The comparison shows that the proposed method improves the active subspace recovery and predictive accuracy, in both the deterministic and probabilistic sense, when only few model observations are available for training, at the cost of increased training time.