Input Dimensionality Reduction Research Articles

Investigation of Surrogate Modeling Options with High-Dimensional Input and Output

Surrogate models are employed in engineering analysis to replace detailed physics-based models to achieve computational efficiency in problems that require multiple evaluations of the model. The accuracy of the surrogate model depends on the quality and quantity of data collected from the expensive model. This paper investigates surrogate modeling options for problems with high-dimensionality in both the input and output spaces. Several methods for reducing the output dimension are investigated, namely, singular value decomposition (SVD), random projection, randomized SVD, and diffusion map; similarly, several methods for input dimension reduction are investigated, namely, variance-based sensitivity analysis and active subspace discovery. The most effective combination of options for input and output dimension reduction is identified in a systematic way, followed by the construction of Gaussian process surrogate models in the low-dimensional space. The prediction error in the original space includes both the reconstruction error and surrogate error; a systematic approach is developed to quantify and compare the relative contributions of the two types of errors. The proposed general, systematic approach of exploring available options is applied to an aircraft fuselage panel. The effectiveness of various dimension reduction techniques with surrogate model construction are investigated in terms of accuracy and computational effort.

A Deep Learning-Based Framework for Low Complexity Multiuser MIMO Precoding Design

Using precoding to suppress multi-user interference is a well-known technique to improve spectra efficiency in multiuser multiple-input multiple-output (MU-MIMO) systems, and the pursuit of high performance and low complexity precoding method has been the focus in the last decade. The traditional algorithms including the zero-forcing (ZF) algorithm and the weighted minimum mean square error (WMMSE) algorithm failed to achieve a satisfactory trade-off between complexity and performance. In this paper, leveraging on the power of deep learning, we propose a low-complexity precoding design framework for MU-MIMO systems. The key idea is to transform the MIMO precoding problem into the multiple-input single-output precoding problem, where the optimal precoding structure can be obtained in closed-form. A customized deep neural network is designed to fit the mapping from the channels to the precoding matrix. In addition, the technique of input dimensionality reduction, network pruning, and recovery module compression are used to further improve the computational efficiency. Furthermore, the extension to the practical MIMO orthogonal frequency-division multiplexing (MIMO-OFDM) system is studied. Simulation results show that the proposed low-complexity precoding scheme achieves similar performance as the WMMSE algorithm with very low computational complexity.

Fault detection and diagnosis to enhance safety in digitalized process system

The increased complexity of digitalized process systems requires advanced tools to detect and diagnose faults early to maintain safe operations. This study proposed a hybrid model that consists of Kernel Principal Component Analysis (kPCA) and DNNs that can be applied to detect and diagnose faults in various processes. The complex data is processed by kPCA to reduce its dimensionality; then, simplified data is used for two separate DNNs for training (detection and diagnosis). The relative performance of the hybrid model is compared with conventional methods. Tennessee Eastman Process was used to confirm the efficacy of the model. The results show the reduction of input dimensionality increases classification accuracy. In addition, splitting detection and diagnosis into two DNNs results in reduced training times and increased classification accuracy. The proposed hybrid model serves as an important tool to detect the fault and take early corrective actions, thus enhancing process safety.

Structure exploiting methods for fast uncertainty quantification in multiphase flow through heterogeneous media

We present a computational framework for dimension reduction and surrogate modeling to accelerate uncertainty quantification in computationally intensive models with high-dimensional inputs and function-valued outputs. Our driving application is multiphase flow in saturated-unsaturated porous media in the context of radioactive waste storage. For fast input dimension reduction, we utilize an approximate global sensitivity measure, for function-valued outputs, motivated by ideas from the active subspace methods. The proposed approach does not require expensive gradient computations. We generate an efficient surrogate model by combining a truncated Karhunen-Loéve (KL) expansion of the output with polynomial chaos expansions, for the output KL modes, constructed in the reduced parameter space. We demonstrate the effectiveness of the proposed surrogate modeling approach with a comprehensive set of numerical experiments, where we consider a number of function-valued (temporally or spatially distributed) QoIs.

Application of Two New Feature Fusion Networks to Improve Real-time Prostate Capsula Detection.

Excess prostate tissue is trimmed near the prostate capsula boundary during transurethral plasma kinetic enucleation of prostate (PKEP) and transurethral bipolar plasmakinetic resection of prostate (PKRP) surgeries. If a large portion of the tissue is removed, a prostate capsula perforation can potentially occur. As such, real-time accurate prostate capsula (PC) detection is critical for the prevention of these perforations. This study investigated the potential for using image denoising, image dimension reduction and feature fusion to improve real-time prostate capsula detection with two objectives. First, this paper mainly studied feature selection and input dimension reduction. Secondly, image denoising was evaluated, as it is of paramount importance to transient stability assessment based on neural networks. Two new feature fusion techniques, maxpooling bilinear interpolation single-shot multibox detector (PBSSD) and bilinear interpolation single shot multibox detector (BSSD) were proposed. Before original images were sent to the neural network, they were processed by principal component analysis (PCA) and adaptive median filter (AMF) for dimension reduction and image denoising. The results showed that the application of PCA and AMF with PBSSD increased the mean average precision (mAP) for prostate capsula images by 8.55% and reached 80.15%, compared with single shot multibox detector (SSD) alone. Application of PCA with BSSD increased the mAP for prostate capsula images by 4.6% compared with SSD alone. Compared with other methods, ours were proven to be more accurate for real-time prostate capsula detection. The improved mAP results suggest that the proposed approaches are powerful tools for improving SSD networks.

Enabling on-device classification of ECG with compressed learning for health IoT

In this paper, an on-device classification of electrocardiography (ECG) with Compressed Learning (CL) for health Internet of Things (IoT) is proposed. A CL algorithm combining with one-dimensional (1-D) Convolutional Neural Network (CNN) that directly learns on ECG signals in the compression domain without expanded normalization is proposed. Such an approach bypasses the reconstruction step and minimizes the raw input data dimension that significantly reduces the processing power. An automatic network optimization framework with Automatic Machine Learning (AutoML) tool Neural Network Intelligence (NNI) is suggested to adapt to the network structure search problem introduced by input dimension reduction with Compression Ratios (CRs). That ensures the minimized model size and operation number under a guaranteed classification accuracy. To implement the resized 1-D CL classifier in hardware, which has different kernel sizes, strides, and output channels under different CRs, a flexible architecture is proposed to further lower power consumption. Evaluated on the MIT-BIH database, the specific 1-D CNN network selected under CR =0.2 achieves a Macro-F1 of 0.9214 on 5-class ECG signal classification, with a 6.4× reduction in FLOPs and a 2.6× decrease in model size with an only 0.028 loss in Macro-F1 compared with the uncompressed situation. Synthesized in UMC 40 nm Low Power process, the hardware architecture with the 1-D CL classifier achieves an energy efficiency of 0.83 μJ/Classification under a 1.1-V power supply at a frequency of 5 MHz.

Line-Segment Feature Extraction Algorithm for Input Dimensionality Reduction in Multi-Layer Perceptron Model

Background/Objectives: Today, the deep neural network has more intermediate layers along with learning data and has gradually become larger to draw a lower result due to the bound conventional neural network. Methods/Statistical analysis: Therefore, in this paper, we proposed the line-segment feature extraction for input dimensionality reduction in pattern mining. The proposed algorithm extracts the line segment information, constituting the image of input data, and assigns a unique value to each segment using predefined filters. Using such unique values to identify the number of line segments, create a one-dimensional vector, in the size of 256 or 512. Findings: This vector is used as the input data for a multi-layer perceptron (MLP). For performance evaluation, LFA was compared with principal component analysis (PCA). As a result, LFA-256 (at 96.81% accuracy) had −0.1% lower performance but a faster speed than PCA (96.91% accuracy), and LFA-512 (at 97.25% accuracy) had 0.35% higher performance than PCA. Improvements/Applications: We will study the recognition service possible to use in industry (mobile devices, PC, edge computing, etc.) and real-life through this algorithm.

Computer Model Emulation with High-Dimensional Functional Output in Large-Scale Observing System Uncertainty Experiments

Observing system uncertainty experiments (OSUEs) have been recently proposed as a cost-effective way to perform probabilistic assessment of retrievals for NASA’s Orbiting Carbon Observatory-2 (OCO-2) mission. One important component in the OCO-2 retrieval algorithm is a full-physics forward model that describes the mathematical relationship between atmospheric variables such as carbon dioxide and radiances measured by the remote sensing instrument. This complex forward model is computationally expensive but large-scale OSUEs require evaluation of this model numerous times, which makes it infeasible for comprehensive experiments. To tackle this issue, we develop a statistical emulator to facilitate large-scale OSUEs in the OCO-2 mission. Within each distinct spectral band, the emulator represents radiance output at irregular wavelengths as a linear combination of basis functions and random coefficients. These random coefficients are then modeled with nearest-neighbor Gaussian processes with built-in input dimension reduction via active subspace. The proposed emulator reduces dimensionality in both input space and output space, so that fast computation is achieved within a fully Bayesian inference framework. Validation experiments demonstrate that this emulator outperforms other competing statistical methods and a reduced order model that approximates the full-physics forward model.

Supervised Learning With First-to-Spike Decoding in Multilayer Spiking Neural Networks.

Experimental studies support the notion of spike-based neuronal information processing in the brain, with neural circuits exhibiting a wide range of temporally-based coding strategies to rapidly and efficiently represent sensory stimuli. Accordingly, it would be desirable to apply spike-based computation to tackling real-world challenges, and in particular transferring such theory to neuromorphic systems for low-power embedded applications. Motivated by this, we propose a new supervised learning method that can train multilayer spiking neural networks to solve classification problems based on a rapid, first-to-spike decoding strategy. The proposed learning rule supports multiple spikes fired by stochastic hidden neurons, and yet is stable by relying on first-spike responses generated by a deterministic output layer. In addition to this, we also explore several distinct, spike-based encoding strategies in order to form compact representations of presented input data. We demonstrate the classification performance of the learning rule as applied to several benchmark datasets, including MNIST. The learning rule is capable of generalizing from the data, and is successful even when used with constrained network architectures containing few input and hidden layer neurons. Furthermore, we highlight a novel encoding strategy, termed “scanline encoding,” that can transform image data into compact spatiotemporal patterns for subsequent network processing. Designing constrained, but optimized, network structures and performing input dimensionality reduction has strong implications for neuromorphic applications.

Learning of classification tasks with an array of uniform-mode spin Hall nano-oscillators

Recently, a system of spintronic vortex oscillators has been experimentally trained to classify vowel sounds. In this paper, we have carried out a combination of device-level and system-level simulations to train a system of spin Hall nano oscillators (SHNOs) of smaller size (25X lower in area compared to those vortex oscillators) for such data classification tasks. Magnetic moments precess in an uniform mode as opposed to the vortex mode in our oscillators. We have trained our system to classify inputs in various popular machine learning data sets like Fisher’s Iris data set of flowers, Wisconsin Breast Cancer (WBC) data set, and MNIST data set of handwritten digits. We have employed a new technique for input dimensionality reduction here so that the clustering/target synchronization pattern changes based on the nature of the data in the different data sets. Our demonstration of learning in a system of such small SHNOs for a wide range of data sets is promising for scaling up the oscillator-based neuromorphic system for complex data classification tasks.

Precise cutterhead torque prediction for shield tunneling machines using a novel hybrid deep neural network

Shield tunneling machine is an important large-scale engineering machine used for tunnel excavation. During the tunneling process, precise cutterhead torque prediction is of vital significance for adjusting operational parameters and avoiding cutterhead jamming, which seriously affects the efficiency, cost and safety of tunneling process. In this paper, a novel hybrid deep neural network (HDNN) is presented for accurately predicting the cutterhead torque for shield tunneling machines based on the equipment operational and status parameters. To begin with, correlation analysis based on cosine similarity between the parameters and the cutterhead torque are conducted for parameter selection and input dimension reduction. Then, selected parameters are fed into the proposed hybrid deep neural network, combing convolutional neural network (CNN) and long short-term memory (LSTM) to extract implicit features and sequential features. On this basis, useful deep information can be fully exploited and utilized for cutterhead torque prediction. Moreover, to further improve the prediction performance and alleviate the gradient disappearing during deep-layer network training, we integrate the residual network module into the proposed neural network. Finally, 15 different datasets constructed from the actual project data are utilized to validate the effectiveness and superiority of the proposed method. The results show that the coincidence degree between the predicted curves and the actual curves of the proposed HDNN is much higher than that of the existing machine learning-based and deep learning-based models. Therefore, the prediction accuracy and the generalization ability of the proposed method outperforms the other data-driven methods. Moreover, on the 15 different datasets with different geological conditions, the highest prediction accuracy of the proposed HDNN is up to 97.4%, and the average prediction accuracy is approximately 96.2%. It can be concluded that the proposed HDNN is capable of accurately predicting the cutterhead torque even under complicated geological conditions, which is provided with high industrial application value.

Relationships between motor and cognitive functions and subsequent post-stroke mood disorders revealed by machine learning analysis

Mood disorders (e.g. depression, apathy, and anxiety) are often observed in stroke patients, exhibiting a negative impact on functional recovery associated with various physical disorders and cognitive dysfunction. Consequently, post-stroke symptoms are complex and difficult to understand. In this study, we aimed to clarify the cross-sectional relationship between mood disorders and motor/cognitive functions in stroke patients. An artificial neural network architecture was devised to predict three types of mood disorders from 36 evaluation indices obtained from functional, physical, and cognitive tests on 274 patients. The relationship between mood disorders and motor/cognitive functions were comprehensively analysed by performing input dimensionality reduction for the neural network. The receiver operating characteristic curve from the prediction exhibited a moderate to high area under the curve above 0.85. Moreover, the input dimensionality reduction retrieved the evaluation indices that are more strongly related to mood disorders. The analysis results suggest a stress threshold hypothesis, in which stroke-induced lesions promote stress vulnerability and may trigger mood disorders.

A Distributed Active Subspace Method for Scalable Surrogate Modeling of Function Valued Outputs

We present a distributed active subspace method for training surrogate models of complex physical processes with high-dimensional inputs and function valued outputs. Specifically, we represent the model output with a truncated Karhunen–Loeve (KL) expansion, screen the structure of the input space with respect to each KL mode via the active subspace method, and finally form an overall surrogate model of the output by combining surrogates of individual output KL modes. To ensure scalable computation of the gradients of the output KL modes, needed in active subspace discovery, we rely on adjoint-based gradient computation. The proposed method combines benefits of active subspace methods for input dimension reduction and KL expansions used for spectral representation of the output field. We provide a mathematical framework for the proposed method and conduct an error analysis of the mixed KL active subspace approach. Specifically, we provide an error estimate that quantifies errors due to active subspace projection and truncated KL expansion of the output. We demonstrate the numerical performance of the surrogate modeling approach with an application example from biotransport.

Line-segment Feature Analysis Algorithm Using Input Dimensionality Reduction for Handwritten Text Recognition

Recently, demand for handwriting recognition, such as automation of mail sorting, license plate recognition, and electronic memo pads, has exponentially increased in various industrial fields. In addition, in the image recognition field, methods using artificial convolutional neural networks, which show outstanding performance, have been applied to handwriting recognition. However, owing to the diversity of recognition application fields, the number of dimensions in the learning and reasoning processes is increasing. To solve this problem, a principal component analysis (PCA) technique is used for dimensionality reduction. However, PCA is likely to increase the accuracy loss due to data compression. Therefore, in this paper, we propose a line-segment feature analysis (LFA) algorithm for input dimensionality reduction in handwritten text recognition. This proposed algorithm extracts the line segment information, constituting the image of input data, and assigns a unique value to each segment using 3 × 3 and 5 × 5 filters. Using the unique values to identify the number of line segments and adding them up, a 1-D vector with a size of 512 is created. This vector is used as input to machine-learning. For the performance evaluation of the method, the Extending Modified National Institute of Standards and Technology (EMNIST) database was used. In the evaluation, PCA showed 96.6% and 93.86% accuracy with k-nearest neighbors (KNN) and support vector machine (SVM), respectively, while LFA showed 97.5% and 98.9% accuracy with KNN and SVM, respectively.

Structural Plasticity on the SpiNNaker Many-Core Neuromorphic System.

The structural organization of cortical areas is not random, with topographic maps commonplace in sensory processing centers. This topographical organization allows optimal wiring between neurons, multimodal sensory integration, and performs input dimensionality reduction. In this work, a model of topographic map formation is implemented on the SpiNNaker neuromorphic platform, running in realtime using point neurons, and making use of both synaptic rewiring and spike-timing dependent plasticity (STDP). In agreement with Bamford et al. (2010), we demonstrate that synaptic rewiring refines an initially rough topographic map over and beyond the ability of STDP, and that input selectivity learnt through STDP is embedded into the network connectivity through rewiring. Moreover, we show the presented model can be used to generate topographic maps between layers of neurons with minimal initial connectivity, and stabilize mappings which would otherwise be unstable through the inclusion of lateral inhibition.

Measuring Motion Capture Data Quality for Data Driven Human Motion Synthesis

In automotive industry, market demands shorter life cycles and individualized products. For manual assembly, this trend leads to more frequent planning of an ever increasing number of process variants. In order to ensure planning quality, virtual verification of manual production is crucial for efficient process optimization. However, virtual verification is not established in practice because available simulation tools require prohibitive manual modeling effort for human motions of acceptable quality. For automating the modeling process, data driven motion synthesis approaches are promising candidates that –however– require high quality input data for acceptable synthesis results. Therefore, objective motion capture data quality measures for data driven human motion synthesis are sought.This work proposes and tests a principal component analysis (PCA) and a Shannon entropy based quality measure. Both measures evaluate post-processed data and thus consider motion capture hardware in combination with a post-processing tool chain.The measures are tested for selectivity and validity using two low cost and two high cost motion capture systems. They differ in selectivity for high and low cost motion capture systems. Both measures correctly predict motion synthesis quality in tests with treadmill walking. Therefore, they can be employed for testing if a motion capture system is suitable for data driven motion synthesis that relies on PCA for input dimension reduction. Further research on robustness of the measures against motion variation is proposed.

English

Epilepsy is a common neurological disorder that is characterized by recurrent unprovoked seizures. About 40 to 50 million people worldwide are reported to have epilepsy. In this paper the authors present clinical decision support system (DSS) for the diagnosis of epilepsy. The DSS is developed by using Multilayer Perceptron (MLP), Generalized Feed Forward Neural Network (GFF-NN) and Elman Neural Network (E-NN). The validity of neural networks to diagnose the epilepsy is checked and the most suitable neural network is recommended for the diagnosis of epilepsy. Also the different feature enhancement techniques like principal component analysis (PCA), FFT and statistical parameters are used for the input dimensionality reduction. Epilepsy diagnosis is modeled as the classification of normal EEG, interictal EEG and ictal EEG. With the different input dimensionality reduction methods performance parameters of MLP, GFF-NN and E-NN are measured and compared. For the GFF-NN, number of free parameter is reduced up to 92.22% when PCA is used for input dimensionality reduction with the overall accuracy of 98.61%.

Optimal MLP neural network classifier for fault detection of three phase induction motor

Induction motors are critical components in commercially available equipments and industrial processes due to cost effective and robust performance. Under various operating stresses, motors deteriorate their conditions which result into various faults. Early detection and diagnosis of these faults are desirable for online condition assessment, product quality assurance and improved operational efficiency. From the related work reported so far it is observed that researchers used vibration analysis, harmonics present in stator current, chemical analysis, electromagnetic analysis, etc. As these approaches are complex in view of the requirement of precise measurement and mathematical modeling. As compared to analytical methods, AI based schemes are more efficient and accurate. In this paper optimal MLP NN based classifier is proposed for fault detection which is inexpensive, reliable, and noninvasive by employing more readily available information such as stator current. Detailed design procedure for MLP and SOM NN models is given for which simple statistical parameters are used as input feature space and Principal Component Analysis is used for reduction of input dimensionality. Robustness of classifier to noise is verified on unseen data by introducing controlled Gaussian and Uniform noise in input and output.

Automatic classification of auditory brainstem responses using SVM-based feature selection algorithm for threshold detection

This paper presents a novel system for automatic recognition of auditory brainstem responses (ABR) to detect hearing threshold. ABR is an important potential signal for determining objective audiograms. Its detection is usually performed by medical experts with often basic signal processing techniques. The proposed system comprises of two stages. In the first stage, for feature extraction, a set of raw amplitude values, a set of discrete cosine transform (DCT) coefficients and a set of discrete wavelet transform (DWT) approximation coefficients are calculated and extracted from signals separately as three different sets of feature vectors. These features are then selected by a modified adaptive method, which mainly supports to the input dimension reduction via selecting the most significant feature components. In the second stage, the feature vectors are classified by a support vector machine (SVM) classifier which is a powerful advanced technique for solving supervised binary classification problem due to its generalization ability. After that the proposed system is applied to real ABR data and it is resulted in a very good sensitivity, specificity and accuracy levels for DCT coefficients such as 99.2%, 94.0% and 96.2%, respectively. Consequently, the proposed system can be used for recognition of ABRs for hearing threshold detection.

A support vector machine classifier algorithm based on a perturbation method and its application to ECG beat recognition systems

In this paper, we introduce a novel system for ECG beat recognition using Support Vector Machine (SVM) classifier designed by a perturbation method. Three feature extraction methods are comparatively examined in reduced dimensional feature space. The dimension of each feature set is reduced by using perturbation method. If there exist redundant data components in training data set, they can be discarded by analyzing the total disturbance of the SVM output corresponding to the perturbed inputs. Thus, the input dimension size is reduced and network becomes smaller. Algorithm for input dimension reduction is first formulated and then applied to real ECG data for recognition of beat patterns. After the preprocessing of ECG data, four types of ECG beats obtained from the MIT-BIH database are recognized with the accuracy of 96.5% by the proposed system together with discrete cosine transform.