Low-dimensional Input Research Articles

Real-time prediction of fuel consumption and emissions based on deep autoencoding support vector regression for cylinder pressure-based feedback control of marine diesel engines

Predictive models serving as virtual sensors for online optimization and feedback control of diesel engines is gaining increasing attention. However, existing prediction models fall short in simultaneously achieving high prediction accuracy and fast computational speed. In this paper, a novel machine learning algorithm called deep autoencoding support vector regression (DASVR) was proposed, which combines the powerful non-linear feature extraction capability of artificial neural network (ANN) with the good adaptability of support vector regression (SVR) to low-dimensional input spaces. ANN-based autoencoder is firstly employed to extract features from the original high-dimensional input space, forming a low-dimensional latent variable space. SVR is then employed to perform non-linear mapping between latent variables and target output variables. Experiments were conducted on a 16-cylinder marine diesel engine under different load, rail pressure, and injection timing conditions. Prediction models for fuel consumption, NOx, PM, HC, and PM emissions were established based on experimental data and DASVR algorithm. The maximum error of DASVR-based models for the five desired output variables is less than 3.8 % under different operating conditions. DAVSR-based predictive models outperform conventional ANN-based and SVR-based models in terms of both prediction accuracy and real-time capability, and can theoretically meet the real-time requirement below 3000 r/min.

Inverse analysis of granular flows using differentiable graph neural network simulator

Inverse problems in granular flows, such as landslides and debris flows, involve estimating material parameters or boundary conditions based on a target runout profile. Traditional high-fidelity simulators for these inverse problems are computationally expensive, restricting the number of possible simulations. These simulators are also non-differentiable, making efficient gradient-based optimization methods for high-dimensional spaces inapplicable. Machine learning-based surrogate models offer computational efficiency and differentiability. However, they often struggle to generalize beyond their training data due to relying on low-dimensional input–output mappings that fail to capture the complete physics of granular flows. We propose a novel differentiable graph neural network simulator (GNS) that combines reverse mode automatic differentiation of graph neural networks with gradient-based optimization for solving inverse problems. GNS learns the dynamics of granular flow by representing the system as a graph and predicts the evolution of the graph at the next timestep, given the current state. The differentiable GNS demonstrates optimization capabilities beyond the training data. We demonstrate the effectiveness of our method for inverse estimation across single and multi-parameter optimization problems, including evaluating material properties and boundary conditions for a target runout distance and designing baffle locations to limit a landslide runout. Our proposed differentiable GNS framework solves inverse problems with orders of magnitude faster convergence than the conventional gradient-based optimization approach using finite difference.

Fusion dynamical systems with machine learning in imitation learning: A comprehensive overview

Imitation Learning (IL), also referred to as Learning from Demonstration (LfD), holds significant promise for capturing expert motor skills through efficient imitation, facilitating adept navigation of complex scenarios. A persistent challenge in IL lies in extending generalization from historical demonstrations, enabling the acquisition of new skills without re-teaching. Dynamical system-based IL (DSIL) emerges as a significant subset of IL methodologies, offering the ability to learn trajectories via movement primitives and policy learning based on experiential abstraction. This paper emphasizes the fusion of theoretical paradigms, integrating control theory principles inherent in dynamical systems into IL. This integration notably enhances robustness, adaptability, and convergence in the face of novel scenarios. This survey aims to present a comprehensive overview of DSIL methods, spanning from classical approaches to recent advanced approaches. We categorize DSIL into autonomous dynamical systems and non-autonomous dynamical systems, surveying traditional IL methods with low-dimensional input and advanced deep IL methods with high-dimensional input. Additionally, we present and analyze three main stability methods for IL: Lyapunov stability, contraction theory, and diffeomorphism mapping. Our exploration also extends to popular policy improvement methods for DSIL, encompassing reinforcement learning, deep reinforcement learning, and evolutionary strategies. The primary objective is to expedite readers’ comprehension of dynamical systems’ foundational aspects and capabilities, helping identify practical scenarios and charting potential future directions. By offering insights into the strengths and limitations of dynamical system methods, we aim to foster a deeper understanding among readers. Furthermore, we outline potential extensions and enhancements within the realm of dynamical systems, outlining avenues for further exploration.

Efficient Algorithms for Non-gaussian Single Index Models with Generative Priors

In this work, we focus on high-dimensional single index models with non-Gaussian sensing vectors and generative priors. More specifically, our goal is to estimate the underlying signal from i.i.d. realizations of the semi-parameterized single index model, where the underlying signal is contained in (up to a constant scaling) the range of a Lipschitz continuous generative model with bounded low-dimensional inputs, the sensing vector follows a non-Gaussian distribution, the noise is a random variable that is independent of the sensing vector, and the unknown non-linear link function is differentiable. Using the first- and second-order Stein's identity, we introduce efficient algorithms to obtain estimated vectors that achieve the near-optimal statistical rate. Experimental results on image datasets are provided to support our theory.

ANOVA-GP Modeling for High-Dimensional Bayesian Inverse Problems

Markov chain Monte Carlo (MCMC) stands out as an effective method for tackling Bayesian inverse problems. However, when dealing with computationally expensive forward models and high-dimensional parameter spaces, the challenge of repeated sampling becomes pronounced. A common strategy to address this challenge is to construct an inexpensive surrogate of the forward model, which cuts the computational cost of individual samples. While the Gaussian process (GP) is widely used as a surrogate modeling strategy, its applicability can be limited when dealing with high-dimensional input or output spaces. This paper presents a novel approach that combines the analysis of variance (ANOVA) decomposition method with Gaussian process regression to handle high-dimensional Bayesian inverse problems. Initially, the ANOVA method is employed to reduce the dimension of the parameter space, which decomposes the original high-dimensional problem into several low-dimensional sub-problems. Subsequently, principal component analysis (PCA) is utilized to reduce the dimension of the output space on each sub-problem. Finally, a Gaussian process model with a low-dimensional input and output is constructed for each sub-problem. In addition to this methodology, an adaptive ANOVA-GP-MCMC algorithm is proposed, which further enhances the adaptability and efficiency of the method in the Bayesian inversion setting. The accuracy and computational efficiency of the proposed approach are validated through numerical experiments. This innovative integration of ANOVA and Gaussian processes provides a promising solution to address challenges associated with high-dimensional parameter spaces and computationally expensive forward models in Bayesian inference.

Industrial Data Modeling With Low-Dimensional Inputs and High-Dimensional Outputs

Data-driven modeling will be complicated for a process if the output quality indices are defined in a high-dimensional space, e.g., a quality distribution. In this work, a novel probabilistic modeling method is proposed for industrial processes with low-dimensional inputs and high-dimensional outputs. First, based on a limited sample set, the variational autoencoder (VAE) is applied to extract features of the high-dimensional outputs. Next, a Gaussian Process (GP) model is established on the sub-manifold space defined by the low-dimensional features, and the high-dimensional predictions can be obtained through the VAE reverse procedure. Finally, a deep composing kernel strategy is developed to capture the nonlinearity and correlation hidden in the features. It can significantly improve the generalization performance of the GP model. The effectiveness of the proposed modeling algorithm is demonstrated by applications in a continuous crystallizer system and an ethylene homo-polymerization system.

Dynamic memristor for physical reservoir computing.

Reservoir computing (RC) has attracted considerable attention for its efficient handling of temporal signals and lower training costs. As a nonlinear dynamic system, RC can map low-dimensional inputs into high-dimensional spaces and implement classification using a simple linear readout layer. The memristor exhibits complex dynamic characteristics due to its internal physical processes, which renders them an ideal choice for the implementation of physical reservoir computing (PRC) systems. This review focuses on PRC systems based on memristors, explaining the resistive switching mechanism at the device level and emphasizing the tunability of their dynamic behavior. The development of memristor-based reservoir computing systems is highlighted, along with discussions on the challenges faced by this field and potential future research directions.

Performance evaluation of KPCA pre-imaging methods for speech signal denoising

<p class="Abstract">Kernel principal component analysis (KPCA) has gained wider interest amongst the researchers in nonlinear dimensionality reduction, data compression, feature extraction, and denoising applications. In KPCA, data from low dimensional input space implicitly mapped to higher dimensional feature space where linear PCA is enforced. However, for denoising application data need to invert back to input space, which is impossible and known as pre-imaging problems. Over recent years, several pre-imaging methods have proposed each with its benefits and disadvantages. In this paper, we evaluated the performance for selected pre-imaging methods for denoising speech signal, whose intelligibility and quality degraded by background noises. Further, we extend our work by comparing the performance of these pre-imaging methods by objective evaluation of denoised speech signal on generated toy examples and NOIZUES database.</p>

DFLOWFS: Deep Fast-Learning Optimal Weight Fuzzy System with bottom-up hierarchical structure for high-dimensional data regression

As a model for reasoning and decision-making based on fuzzy rules, fuzzy systems have high interpretability. However, when the data dimension increases, the fuzzy system will face the problem of “rule explosion”, making it difficult to learn and predict effectively. In this paper, the fuzzy system trained by the FLOWFS (Fast-Learning with Optimal Weights for Fuzzy Systems) algorithm is used as sub-module in the deep fuzzy system, and the deep fuzzy system DFLOWFS (Deep FLOWFS) is constructed from the bottom-up hierarchical structure as the following three steps. 1) The FLOWFS algorithm assigns weight attributions to each fuzzy rule, and the rule weights are trained by the least square method with regularization terms to shorten training time and improve accuracy. 2) Three strategies of dividing high-dimensional inputs into multiple low-dimensional inputs are proposed as sequential division, random division and correlation division. Then, it is verified by experiments that the correlation division has the best performance. 3) The sub-module discarding method is proposed to discard the sub-modules with poor performance to have a maximum improvement of 13.8% compared to the DFLOWFS without using the sub-module discarding method. Then, the optimized DFLOWFS is verified and compared with the other three classic regression models on the three UCI datasets. Experiments show that with the increase of the data dimension, DFLOWFS not only have good interpretability but also have good accuracy. Furthermore, DFLOWFS performs best among all models in comprehensive scores, with good learning ability and generalization ability. Therefore, the proposed strategies with hierarchical structure for optimal shallow fuzzy systems are effective, which give a new insight for fuzzy system research.

Bayesian Global Optimization Gated Recurrent Unit Model for Human-Driven Vehicle Lane-Change Trajectory Prediction Considering Hyperparameter Optimization

To address the low accuracy and inefficiency of current lane-change trajectory prediction methods for human-driven vehicles, this study develops a neural network lane-change trajectory prediction model with hyperparametric optimization capability using Bayesian optimization and gated recurrent units to consider the effect of lane-change intention on vehicle lane-change behavior and to predict it. The proposed model was instantiated using trajectory data of 8,721 vehicles. The results show that the overall recognition accuracy of intention recognition under the optimal input is 93.54%, and the recognition accuracy of keeping straight, left lane-change and right lane-change is 95.59%, 91.72%, and 93.30%, respectively. The root mean square errors of the predicted and actual trajectories to the left and to the right under the optimal input are 0.2582 and 0.2957, respectively. This paper demonstrates that, for the intention recognition module, the low-dimensional input enables the model to obtain high prediction accuracy, while for the trajectory prediction module, the high-latitude input enables the model to obtain a low prediction error. The developed trajectory prediction model can be used to assist in driving decision-making, path planning, and so forth.

Lite and Efficient Deep Learning Model for Bearing Fault Diagnosis Using the CWRU Dataset

Bearing defects are a common problem in rotating machines and equipment that can lead to unexpected downtime, costly repairs, and even safety hazards. Diagnosing bearing defects is crucial for preventative maintenance, and deep learning models have shown promising results in this field. On the other hand, the high complexity of these models can lead to high computational and data processing costs, making their practical implementation challenging. Recent studies have focused on optimizing these models by reducing their size and complexity, but these methods often compromise classification performance. This paper proposes a new approach that reduces the dimensionality of input data and optimizes the model structure simultaneously. A much lower input data dimension than that of existing deep learning models was achieved by downsampling the vibration sensor signals used for bearing defect diagnosis and constructing spectrograms. This paper introduces a lite convolutional neural network (CNN) model with fixed feature map dimensions that achieve high classification accuracy with low-dimensional input data. The vibration sensor signals used for bearing defect diagnosis were first downsampled to reduce the dimensionality of the input data. Next, spectrograms were constructed using the signals of the minimum interval. Experiments were conducted using the vibration sensor signals from the Case Western Reserve University (CWRU) dataset. The experimental results show that the proposed method could be highly efficient in terms of computation while maintaining outstanding classification performance. The results show that the proposed method outperformed a state-of-the-art model for bearing defect diagnosis under different conditions. This approach is not limited to the field of bearing failure diagnosis, but could be applied potentially to other fields that require the analysis of high-dimensional time series data.

Extended sample-based approach for efficient sensitivity analysis of group of random variables

This paper proposes an extended sample-based approach for efficient sensitivity analysis of the group of random variables by extending the original sample-based approach proposed by Jia and Taflanidis. The original sample-based approach can efficiently estimate the sensitivity indices (i.e., Sobol’ indices) for individual and low-dimensional inputs by using kernel density estimation (KDE). However, due to the curse of dimensionality of KDE, it is challenging to directly apply the original sample-based approach to estimate sensitivity indices for high-dimensional groups of inputs. To address the challenges, the proposed approach converts estimating the sensitivity index for groups of inputs with potentially high dimensionality to estimating the sensitivity indices for individual inputs in the group and their lower-order (e.g., second or third) interactions using the idea of variance decomposition, thus limiting the dimension of the required KDE to be low (e.g., only two or three) if higher-order interactions are negligible. Different estimators for the main and total sensitivity indices are proposed to reduce the error accumulation and improve estimation accuracy. Overall, the proposed approach can be used to efficiently estimate the sensitivity indices (including main and total sensitivity indices) for both individual and groups of inputs when higher-order interactions have negligible contributions. The good efficiency and accuracy of the proposed approach are demonstrated through three benchmark examples.

On a nonlinear extension of the principal fitted component model

We propose a nonlinear sufficient dimension reduction method called the kernel principal fitted component model using the kernel method under a reproducing kernel Hilbert space. The kernel principal fitted component is a nonlinear extension of the principal fitted component model, and it is found in the theory of mapping low dimensional input space to the higher dimensional feature space so that we can apply well-developed linear methods to the nonlinear dataset. We derive our method coincides with the generalized sliced inverse regression under some mild assumptions and show the dimension reduction subspace extracted from the kernel principal fitted component model is contained in the central class. In the numerical experiments, we present the kernel principal fitted component model with the Gaussian kernel can extract the linear and nonlinear features well for the models from both forward and inverse regression settings. By applying our method to ovarian cancer microarray dataset, we demonstrate the kernel principal fitted component can provide a competitive prediction accuracy and computational efficiency in the high-dimensional classification problem.

Grassmannian learning mutual subspace method for image set recognition

This paper addresses the problem of object recognition given a set of images as input (e.g., multiple camera sources and video frames). Convolutional neural network (CNN)-based frameworks do not exploit these sets effectively, processing a pattern as observed, not capturing the underlying feature distribution as it does not consider the variance of images in the set. To address this issue, we propose the Grassmannian learning mutual subspace method (G-LMSM), a NN layer embedded on top of CNNs that can process image sets more effectively and can be trained in an end-to-end manner. The image set is first represented by a low-dimensional input subspace and then this input subspace is matched with dictionary subspaces by a similarity of their canonical angles, an interpretable and easy to compute metric. The key idea of G-LMSM is that the dictionary subspaces are learned as points on the Grassmann manifold, optimized with Riemannian stochastic gradient descent. This learning is stable, efficient and theoretically well-grounded. We demonstrate the effectiveness of our proposed method on hand shape recognition, face identification, and facial emotion recognition.

An efficient Kriging model-based importance sampling method for estimating the failure probability-based parameter global sensitivity index with uncertain distribution parameters

The failure probability-based parameter global sensitivity index measures the importance of each uncertain distribution parameter on the uncertainty of failure probability. The crude Monte Carlo simulation for estimating the failure probability-based parameter global sensitivity index involves a complicated nested triple-loop process, and the huge amount of computational burden restricts the application of the failure probability-based parameter global sensitivity. For efficiently analyzing the failure probability-based parameter global sensitivity index, this paper proposes an efficient Kriging model-based importance sampling method. First, the standard normal auxiliary variables are inducted by using the equivalent probability transformation, in which the equivalent limit state function with respect to the uncertain distribution parameters and the standard normal auxiliary variables are constructed, so that the importance sampling samples of uncertain distribution parameters and the standard normal auxiliary variables can be generated based on the equivalent limit state function. Secondly, based on the importance sampling technique, the formula of single-loop method for estimating the failure probability-based parameter global sensitivity index is derived in which the importance sampling samples can be reused to obtain all uncertain distribution parameters' failure probability-based global sensitivity indices. Thirdly, the equivalent probability transformation is used again to transform the importance sampling samples in the high-dimensional space into the relatively low-dimensional input space, and the Kriging model of the original limit state function with relatively low-dimensional input space is adaptively constructed to recognize the states (failure or safety) of all high-dimensional importance sampling samples. Then, the failure probability-based parameter global sensitivity indices can be efficiently estimated. The results of three case studies verify the accuracy and efficiency of the proposed method.

Verification of Image-Based Neural Network Controllers Using Generative Models

Although neural networks are effective tools for processing information from image-based sensors to produce control actions, their complex nature limits their use in safety-critical systems. For this reason, recent work has focused on combining techniques in formal methods and reachability analysis to obtain guarantees on the closed-loop performance of neural network controllers. However, these techniques do not scale to the high-dimensional and complicated input space of image-based neural network controllers. This work proposes a method to address these challenges by training a generative adversarial network to map states to plausible input images. Concatenating the generator network with the control network results in a network with a low-dimensional input space, which allows for the use of existing closed-loop verification tools to obtain formal guarantees on the performance of image-based controllers. This approach is applied to provide safety guarantees for an image-based neural network controller for an autonomous aircraft taxi problem. The resulting guarantees are with respect to the set of input images modeled by the generator network, and so a recall metric is provided to evaluate how well the generator captures the space of plausible images.

Image fairness in deep learning: problems, models, and challenges

In recent years, it has been revealed that machine learning models can produce discriminatory predictions. Hence, fairness protection has come to play a pivotal role in machine learning. In the past, most studies on fairness protection have used traditional machine learning methods to enforce fairness. However, these studies focus on low dimensional inputs, such as numerical inputs, whereas more recent deep learning technologies have encouraged fairness protection with image inputs through deep model methods. These approaches involve various object functions and structural designs that break the spurious correlations between targets and sensitive features. With these connections broken, we are left with fairer predictions. To better understand the proposed methods and encourage further development in the field, this paper summarizes fairness protection methods in terms of three aspects: the problem settings, the models, and the challenges. Through this survey, we hope to reveal research trends in the field, discover the fundamentals of enforcing fairness, and summarize the main challenges to producing fairer models.

Enhanced SCADA IDS Security by Using MSOM Hybrid Unsupervised Algorithm

In Self-Organizing Maps (SOM) are unsupervised neural networks that cluster high dimensional data and transform complex inputs into easily understandable inputs. To find the closest distance and weight factor, it maps high dimensional input space to low dimensional input space. The Closest node to data point is denoted as a neuron. It classifies the input data based on these neurons. The reduction of dimensionality and grid clustering using neurons makes to observe similarities between the data. In our proposed Mutated Self Organizing Maps (MSOM) approach, we have two intentions. One is to eliminate the learning rate and to decrease the neighborhood size and the next one is to find out the outliers in the network. The first one is by calculating the median distance (MD) between each node with its neighbor nodes. Then those median values are compared with one another. In case, if any of the MD values significantly varies from the rest then it is declared as anomaly nodes. In the second phase, we find out the quantization error (QE) in each instance from the cluster center.

ENHANCED SCADA IDS SECURITY BY USING MSOM HYBRID UNSUPERVISED ALGORITHM

In Self-Organizing Maps (SOM) are unsupervised neural networks that cluster high dimensional data and transform complex inputs into easily understandable inputs. To find the closest distance and weight factor, it maps high dimensional input space to low dimensional input space. The Closest node to data point is denoted as a neuron. It classifies the input data based on these neurons. The reduction of dimensionality and grid clustering using neurons makes to observe similarities between the data. In our proposed Mutated Self Organizing Maps (MSOM) approach, we have two intentions. One is to eliminate the learning rate and to decrease the neighborhood size and the next one is to find out the outliers in the network. The first one is by calculating the median distance (MD) between each node with its neighbor nodes. Then those median values are compared with one another. In case, if any of the MD values significantly varies from the rest then it is declared as anomaly nodes. In the second phase, we find out the quantization error (QE) in each instance from the cluster center.

Wind Power Prediction Based on a Hybrid Granular Chaotic Time Series Model

For realizing high-accuracy short-term wind power prediction, a hybrid model considering physical features of data is proposed in this paper, with consideration of chaotic analysis and granular computing. First, considering the chaotic features of wind power time series physically, data reconstruction in chaotic phase space is studied to provide a low-dimensional input with more information in modeling. Second, considering that meteorological scenarios of wind development are various, complicated, and uncertain, typical chaotic time series prediction models and wind scenarios are analyzed correspondingly via granular computing (GrC). Finally, through granular rule-based modeling, a hybrid model combining reconstructed wind power data and different models is constructed for short-term wind power prediction. Data from real wind farms is taken for experiments, validating the feasibility and effectiveness of the proposed wind power prediction model.