High-dimensional Nonlinear Data Research Articles

Augmented projection Wasserstein distances: Multi-dimensional projection with neural surface

The Wasserstein distance is a fundamental tool for comparing probability distributions and has found broad applications in various fields, including image generation using generative adversarial networks. Despite its useful properties, the performance of the Wasserstein distance decreases when data is high-dimensional, known as the curse of dimensionality. To mitigate this issue, an extension of the Wasserstein distance has been developed, such as the sliced Wasserstein distance using one-dimensional projection. However, such an extension loses information on the original data, due to the linear projection onto the one-dimensional space. In this paper, we propose novel distances named augmented projection Wasserstein distances (APWDs) to address these issues, which utilize multi-dimensional projection with a nonlinear surface by a neural network. The APWDs employ a two-step procedure; it first maps data onto a nonlinear surface by a neural network, then linearly projects the mapped data into a multidimensional space. We also give an algorithm to select a subspace for the multi-dimensional projection. The APWDs are computationally effective while preserving nonlinear information of data. We theoretically confirm that the APWDs mitigate the curse of dimensionality from data. Our experiments demonstrate the APWDs’ outstanding performance and robustness to noise, particularly in the context of nonlinear high-dimensional data.

MADE: Multicurvature Adaptive Embedding for Temporal Knowledge Graph Completion.

Temporal knowledge graphs (TKGs) are receiving increased attention due to their time-dependent properties and the evolving nature of knowledge over time. TKGs typically contain complex geometric structures, such as hierarchical, ring, and chain structures, which can often be mixed together. However, embedding TKGs into Euclidean space, as is typically done with TKG completion (TKGC) models, presents a challenge when dealing with high-dimensional nonlinear data and complex geometric structures. To address this issue, we propose a novel TKGC model called multicurvature adaptive embedding (MADE). MADE models TKGs in multicurvature spaces, including flat Euclidean space (zero curvature), hyperbolic space (negative curvature), and hyperspherical space (positive curvature), to handle multiple geometric structures. We assign different weights to different curvature spaces in a data-driven manner to strengthen the ideal curvature spaces for modeling and weaken the inappropriate ones. Additionally, we introduce the quadruplet distributor (QD) to assist the information interaction in each geometric space. Ultimately, we develop an innovative temporal regularization to enhance the smoothness of timestamp embeddings by strengthening the correlation of neighboring timestamps. Experimental results show that MADE outperforms the existing state-of-the-art TKGC models.

Prototype Enhancement-Based Incremental Evolution Learning for Urban Garbage Classification

Deep neural network (DNN) based on incremental learning provides support for efficient garbage classification tasks. However, it is always challenging to accurately learn and preserve the information of known classes for updating DNN while new tasks are continuously emerging, which also affects the generalization performance of the model. To solve these issues, an incremental evolution learning (IEL) method based on prototype enhancement is proposed to accurately preserve data and improve the model generalization ability. First, a prototype enhancement method based on multi-dimensional Gaussian kernel density estimation is designed, which extends the prototype of each sample based on high-dimensional nonlinear data distribution. Then, the prototype enhancement accurately represents the known class data. Second, a contrastive feature method is proposed to constrain the consistency of features between tasks, which reduces the deviation between different tasks. Then, the extraction preference caused by the class sample imbalance is balanced and the generalization ability is improved. Third, the proposed IEL is used for garbage classification with imbalanced class samples. IEL implements garbage classification by effectively adapting to the differences between known classes and new classes. Finally, experiments on four standard datasets and one public garbage dataset verify that IEL has a strong classification ability in the learning tasks with the increasing number of classes.

BOF steelmaking endpoint carbon content and temperature soft sensor based on supervised dual-branch DBN

Accurate prediction of endpoint carbon content and temperature is critical in the basic oxygen furnace (BOF) steelmaking process. Although deep learning soft sensor approaches have the capacity to extract abstract features from high-dimensional nonlinear steelmaking data, they confront the challenge of a low correlation between acquired features and labels. This work presents a BOF steelmaking soft sensor model based on supervised dual-branch deep belief network (SD-DBN) to address this issue. The SD-DBN model incorporates label information into the feature extraction process and fuses crucial feature information to complete the feature extraction in order to extract features that are closely connected to the target variables. First, the supervised Restricted Boltzmann Machine (RBM) is improved by using a pruning strategy to extract features that are highly correlated with quality information, and then the autocorrelation key feature extraction module is spliced and fused to form a dual-branch feature extraction module to improve key information extraction. Second, stacking the supervised dual-branch RBM modules to build a deep feature extraction network enhances the deep extraction capabilities of data features. This deep network stacking not only increases the impact of essential target data in hierarchical training, but it also acquires characteristics associated with the target variables.

A Projection Pursuit Dynamic Cluster Model for Tourism Safety Early Warning and Its Implications for Sustainable Tourism

According to the United Nations World Tourism Organization, tourism promotes sustainable economic development. Ensuring tourism safety is an essential prerequisite for its sustainable development. In this paper, based on the three evaluation index systems for tourism safety early warning and the collected sample data, we establish three projection pursuit dynamic cluster (PPDC) models by applying group search optimization, a type of swarm intelligence algorithm. Based on case studies, it is confirmed that the results derived from the PPDC models are consistent with the expert judgments. The importance of the evaluation indicators can be sorted and classified according to the obtained optimal projection pursuit vector coefficients, and the tourism risks of the destinations can be ranked according to the sample projection values. Among the three aspects influencing tourism safety in case one, the stability of the tourism destination has the most significant impact, followed by the frequency of disasters. Of the ten evaluation indicators, the frequency of epidemic disease affects tourism safety the most, and the unemployment ratio affects it the second most. Overall, the PPDC model can be adopted for tourism safety early warning with high-dimensional non-linear and non-normal distribution data modeling, as it overcomes the “curse of dimensionality” and the limitations associated with small sample sizes.

Res-TranBiLSTM: An intelligent approach for intrusion detection in the Internet of Things

The Internet of Things (IoT), as the information carrier of the Internet and telecommunications networks, is a new network technology comprising physical entities embedded with electronic components, software and sensors, and characterized by strong complexity and openness. With the massive amount of data, the occurrence of network intrusion is also increasingly frequent, involving industrial control systems, IoT devices, mobile security, cloud services, and telecommunications services. With the diversification and intelligence of cyberattack behaviors, traditional intrusion detection systems (IDSs) face problems—such as insufficient feature extraction and inaccurate model classification—when faced with high-dimensional features and nonlinear massive data. Due to their powerful data representation learning ability, deep learning methods save substantial time in processing high-dimensional and complex intrusion data. On this basis, we propose an intrusion detection model using ResNet, Transformer and BiLSTM (Res-TranBiLSTM) that takes into account both the spatial and temporal features of network traffic. We use the Synthetic Minor Overriding Technique (SMOTE) – Edited Nearest Neighbor (ENN) method to alleviate the degree of data imbalance. In addition, we respectively establish a spatial feature extraction model based on ResNet and a temporal feature extraction model based on Transformer and BiLSTM to extract spatial features and temporal features parallelly. Finally, spatiotemporal features are included to achieve attack detection and classification. Further, simulation experiments are conducted using the public data sets NSL-KDD and CIC-IDS2017. The experiments are implemented using Python programming language and Pytorch framework. The results reveal that the performance of our proposed model is better than that of other models, with accuracy reaching 90.99%, 99.15% and 99.56%, on NSL-KDD dataset, CIC-IDS2017 dataset and MQTTset dataset, respectively. It increased the detection accuracy by about 1%-10% on NSL-KDD dataset and about 0.2%-10% on CIC-IDS2017 dataset, and about 1%-10% on MQTTset dataset. These results demonstrate that this method is effective in constructing and optimizing large-scale IDS in the IoT environment.

A data-driven methodology for predicting residual strength of subsea pipeline with double corrosion defects

The accurate residual strength prediction can support the maintenance planning of damaged subsea pipeline. Recently, machine learning techniques become a remarkable alternative for failure prediction of subsea pipelines. This study presents a data-driven methodology for predicting residual strength of subsea pipeline with double corrosion defects. Firstly, a finite element model of corroded subsea pipeline is established to estimate its residual strength. The effect of corrosion location, size and pipeline parameters on pipeline residual strength are investigated to generate the database. Subsequently, a manifold learning algorithm, i.e., Locally Linear Embedding, is applied to data processing, which can implement feature extraction for high-dimensional nonlinear manifold data. Eventually, the processed data is input into Bayesian regularized artificial neural network algorithm to develop an intelligent model for predicting residual strength of subsea pipeline with double corrosion defects. Compared with the existing algorithms, the prediction results of the developed model indicate that the proposed model has superiorities in error and regression performance. (MSE = 0.0886; R2 = 0.9993).

An Efficient GAN-Based Multi-classification Approach for Financial Time Series Volatility Trend Prediction

Deep learning has achieved tremendous success in various applications owing to its robust feature representations of complex high-dimensional nonlinear data. Financial time-series prediction is no exception. Hence, the volatility trend prediction in financial time series (FTS) has been an active topic for several decades. Inspired by generative adversarial networks (GAN), which have been studied extensively in image processing and have achieved excellent results, we present the ordinal regression GAN for financial volatility trends (ORGAN-FVT) method for the end-to-end multi-classification task of FTS. An improved generative model based on convolutional long short-term memory (ConvLSTM) and multilayer perceptron (MLP) is proposed to capture temporal features effectively and mine the data distribution of volatility trends (short, neutral, and long) from given FTS data. Meanwhile, ordinal regression is leveraged for the discriminator to improve the multi-classification performance, making the model more practical. Finally, we empirically compare ORGAN-FVT with several state-of-the-art approaches on three real-world stock datasets: MICROSOFT(MSFT), Tesla(TSLA), and The People’s Insurance Company of China(PAICC). ORGAN-FVT demonstrated significantly better AUC and F1 scores, at most 20.81% higher than its competitors.

Self-Attention Transformer-Based Architecture for Remaining Useful Life Estimation of Complex Machines

Meaningful feature extraction from multivariate time-series data is still challenging since it takes into account the correlation between pairs of sensors as well as the temporal information of each time-series. Meanwhile, the huge industrial system has evolved into a data-rich environment, resulting in the rapid development and deployment of deep learning for machine RUL prediction. RUL (Remaining Useful Life) examines a system's behavior over the course of its lifetime, that is, from the last inspection to when the system's performance deteriorates beyond a certain point. RUL has been addressed using Long-Short-Term Memory (LSTM) and Convolution Neural Network (CNN), particularly in complex tasks involving high-dimensional nonlinear data. The main focus, however, has been on degradation data. In 2021, a new realistic run-to-failure turbofan engine degradation dataset was released, which differs significantly from the simulation dataset. The key difference is that each cycle's flight duration varies, so the existing deep technique will be ineffective at predicting the RUL for real-world degradation data. We present a Self-Attention Transformer-Based Encoder model to address this problem. The encoder with the time-stamp encoder layer works in parallel to extract features from various sensors at various time stamps. Self-attention enables efficient processing of extended sequences and focuses on key elements of the input time series. Self-attention is used in the proposed Transformer model to access global characteristics from diverse time-series representations. Under real-world flight conditions, we conduct tests on turbofan engine degradation data using variable-length input. The proposed approach for estimating RUL of turbofan engines appears to be efficient based on empirical results.

Novel Feature Selection Method for Nonlinear Support Vector Regression

The development of sparse techniques presents a major challenge to complex nonlinear high-dimensional data. In this paper, we propose a novel feature selection method for nonlinear support vector regression, called FS-NSVR, which first attempts to solve the nonlinear feature selection problem in the regression technology field. FS-NSVR preserves the representative features selected in the complex nonlinear system due to its use of a feature selection matrix in the original space. FS-NSVR is a challenging mixed-integer programming problem that is solved efficiently by using an alternate iterative greedy algorithm. Experimental results on three artificial datasets and five real-world datasets confirm that FS-NSVR effectively selects representative features and discards redundant features in a nonlinear system. FS-NSVR outperforms L1-norm support vector regression, L1-norm least squares support vector regression, and Lp-norm support vector regression on both feature selection ability and regression efficiency.

An online approach for robust parameter design with incremental Gaussian process

Robust parameter design (RPD), an important method for quality improvement, can effectively mitigate the negative impact of fluctuations on product quality. Traditional RPD adopts offline design, that is, the optimal level of parameter combination is fixed by one-time modeling throughout the production process. This strategy is obviously unreasonable. Online RPD breaks through the limitation of traditional offline design, which can update the optimal setting by utilizing the new sample when the current optimal setting of controllable factors is not suitable. However, there are still some problems in the current version of online RPD, such as poor data fitting ability of response surface model and low efficiency of parameter design. In this paper, a new online RPD method based on Gaussian process (GP) is proposed. The GP model is used to construct the response surface, which has the capacity of dealing with high-dimensional nonlinear data. But traditional GP method adopts batch learning, it cannot update the model online with new samples. So this paper proposes an incremental Gaussian process model (IGP), which can update the response surface in real-time. In the proposed IGP based online robust parameter design method (IGP-RPD), an effective optimization strategy is used to find the optimal setting of controllable factors, and a reasonable selection criterion is used to determine the noise factor setting for the next stage. The optimal setting of the controllable factor in the previous stage and the currently observed noise factor are used as input, and the corresponding quality characteristic is taken as the output. The input and output form a new sample to update the response surface model. In this way, the RPD process can be redone continuously until the desirable optimal setting of the controllable factor is found. Three cases are used to verify the IGP-RPD method and compare it with the existing methods. The experiments manifest that the IGP-RPD method has better performance in both accuracy and efficiency.

Geometrically interpretable Variance Hyper Rectangle learning for pattern classification

Many current intrinsically interpretable machine learning models can only handle the data that are linear, low-dimensional, and relatively independent attributes and often with discrete attribute values, while the models that are capable of handling high-dimensional nonlinear data, like deep learning, have very poor interpretability. Based on the geometric characteristics, a new idea of accurately wrapping the data region with minimum-volume geometry is proposed for pattern classification. The Variance Hyper Rectangle (VHR) model presented in this paper is a realization of the idea. The VHR model uses the minimum-volume hyper rectangles, obtained through projection variance calculation, to wrap the regions occupied by a category of data, hence it has strong and clear geometric interpretability. In addition, the VHR model is well suited for large data volume, as it approaches the linear complexity in both time and space. Extensive qualitative and quantitative experiments are performed on seven real-world data sets, demonstrating that VHR outperforms the state-of-the-art interpretable methods while running quickly.

Risk Prediction of Coal and Gas Outburst in Deep Coal Mines Based on the SAPSO-ELM Algorithm.

Effective risk prevention and management in deep coal mines can reduce the occurrences of outburst accidents and casualties. To address the low accuracy and inefficiency of coal–gas outburst prediction in deep coal mines, this study proposes a deep coal–gas outburst risk prediction method based on kernal principal component analysis (KPCA) and an improved extreme learning machine (SAPSO-ELM) algorithm. Firstly, high-dimensional nonlinear raw data were processed by KPCA. Secondly, the extracted sequence of outburst-causing indicator principal components were used as the input variables for the simulated annealing particle swarm algorithm (SAPSO), which was proposed to optimize the input layer weights and implied layer thresholds of the ELM. Finally, a coal and gas outburst risk prediction model for a deep coal mine based on the SAPSO-ELM algorithm was developed. The research results show that, compared with the ELM and PSO-ELM algorithms, the SAPSO-ELM optimization algorithm significantly improved the accuracy of risk prediction for coal–gas outbursts in deep coal mines, and the accuracy rate was as high as 100%. This study enriches the theory and methods of safety management in deep coal mines, and effectively helps coal mine enterprises in improving their ability to manage coal–gas outburst risks.

Additive Noise Model Structure Learning Based on Spatial Coordinates

A new algorithm named SCB (Spatial Coordinates Based) algorithm is presented for structure learning of additive noise model, which can effectively deal with nonlinear arbitrarily distributed data. This paper makes three specific contributions. Firstly, SC (Spatial Coordinates) coefficient is proposed to use as a standard of independence test and CSC (Conditional Spatial Coordinates) coefficient as a standard of conditional independence test. Secondly, it is proved that the CSC coefficient conforms to the standard normal distribution and the HSIC independence test can be regarded as a special case of the SC coefficient. Finally, based on the SC coefficient, the SCB algorithm is proposed, and the experimental comparison with some existing algorithms on seven classical networks shows that the SCB algorithm has better performance. In particular, SCB algorithm can deal with large sample, high dimensional nonlinear data, and maintain good accuracy and time performance.

A multi-step forecasting model of online car-hailing demand

Multi-step forecasting of online car-hailing demand is necessary for the long-term planning of traffic resources. However, accurate forecasting is very challenging, because it is difficult to extract the dynamic spatial-temporal dependencies from high-dimensional nonlinear data. This paper proposes a regional traffic demand forecasting framework (RTDFF) to predict the boarding and alighting demand of online car-hailing. The RTDFF can learn robust representations of traffic patterns with multi-resolution spatial dependencies. It uses a 3D deconvolution component to enhance the spatial resolution of intra-regions and a 2D residual component to connect the large-scale spatial dependence. This model investigates the effect of the 3D convolution in capturing the long-term temporal dependence. Hybrid modeling is conducted with multi-granularity temporal segmentation and fusion. The experiments were carried out using real-world datasets. The results verify that RTDFF is accurate and stable, and that it has slower deterioration than baseline models, with the raise of prediction steps.

Fuzzy vector quantization with a step-optimizer to improve pattern classification

The paper proposes a novel supervised algorithm for data classification based on fuzzy vector quantization. The novelty here lies in including an optimization procedure to determine the step size of the quantization process. Adoption of fuzzy space for data exploration eliminates the problem of class-overlapping, while dealing with high dimensional non-linear data. In the proposed approach, the input data are first been transformed from the Euclidean space to the fuzzy space with three membership grades. Next, we have used uniform manifold approximation algorithm (UMAP), for projecting the data in visual space by selection of the most contrasting features from the fuzzy vectors. Then these features are passed through a quantization step with a novel step-optimization technique. Optimizing the quantization step and making it independent of data sets significantly speeds up the process. As the class information of all the feature vectors obtained by UMAP is known, majority voting principle has been used to locate the class-centroids in the subsequent step which in turn represent the class labels of the test samples. In test phase, after obtaining the test vectors, the Hyperspherical Direction Cosines (HDC) between the test vectors and the previously obtained class-centroids are evaluated. The test sample is finally assigned that class label where the sum of absolute differences (SAD) of these direction cosines is minimum. We have validated our classifier on various benchmark data sets and achieved higher accuracy with significantly low computation time than the existing state-of-the-art algorithms.

Practical CO2—WAG Field Operational Designs Using Hybrid Numerical-Machine-Learning Approaches

Machine-learning technologies have exhibited robust competences in solving many petroleum engineering problems. The accurate predictivity and fast computational speed enable a large volume of time-consuming engineering processes such as history-matching and field development optimization. The Southwest Regional Partnership on Carbon Sequestration (SWP) project desires rigorous history-matching and multi-objective optimization processes, which fits the superiorities of the machine-learning approaches. Although the machine-learning proxy models are trained and validated before imposing to solve practical problems, the error margin would essentially introduce uncertainties to the results. In this paper, a hybrid numerical machine-learning workflow solving various optimization problems is presented. By coupling the expert machine-learning proxies with a global optimizer, the workflow successfully solves the history-matching and CO2 water alternative gas (WAG) design problem with low computational overheads. The history-matching work considers the heterogeneities of multiphase relative characteristics, and the CO2-WAG injection design takes multiple techno-economic objective functions into accounts. This work trained an expert response surface, a support vector machine, and a multi-layer neural network as proxy models to effectively learn the high-dimensional nonlinear data structure. The proposed workflow suggests revisiting the high-fidelity numerical simulator for validation purposes. The experience gained from this work would provide valuable guiding insights to similar CO2 enhanced oil recovery (EOR) projects.

Stock price forecasting based on Hausdorff fractional grey model with convolution and neural network.

Forecast of stock prices can guide investors' investment decisions. Due to the high-dimensional and long-memory characteristics of stock data, it is difficult to predict. The fractional grey model with convolution (FGMC (1, m)) can be used to predict time series, because of its memory and ability to process high-dimensional data. However, the FGMC (1, m) model has some disadvantages, including complex calculation, loss of information, and approximate background values. In this paper, Hausdorff fractional derivative and Newton-Cotes formula are used to optimize these shortcomings and can get a Hausdorff fractional grey model with convolution (HFGMC (1, m)) model. The HFGMC (1, m)-LLE-BP model is proposed in this paper. HFGMC (1, m) provides a solution that can reduce the complexity of the cumulative generator matrix calculation and preserve the global information of the sequence. Newton-Cotes formula is used to calculate the background value, which can solve the shortcomings of approximate background values. The HFGMC (1, m) model is used to predict the linear component of the sequence, and the BP neural network is used to predict the nonlinear component of the sequence. In addition, because of the high-dimensional and nonlinear characteristics of stock data, a local linear embedding (LLE) algorithm is used to remove redundant information in high-dimensional non-linear data. The experimental results show that the HFGMC (1, m)-LLE-BP model is effective for predicting the stock price in different trends.

Quality evaluation of enterprise environmental accounting information disclosure based on projection pursuit model

To strengthen China’s supervision of the quality of environmental accounting information disclosure and improve this quality, this paper established a set of evaluation systems. Based on the principles of “relevance” and “reliability”, 28 evaluation indicators were selected, and a projection pursuit model was innovatively introduced according to the characteristics of the evaluation data in this study. This model can not only overcome the artificially weighted interference and restrictions on the data structure of the traditional evaluation method but also can achieve better robustness, anti-interference ability and accuracy when processing high-dimensional nonlinear data. To test whether the method is simple and effective, the study selected 34 thermal power listed companies in China as a sample. The evaluation results of the key quantitative indicators showed that the disclosure level of the whole sample is low, which indicates that the information disclosed by each enterprise is not substantial. After verification, the results are basically consistent with the original data. Finally, this paper analyzed the problems in the disclosure of the evaluation results and proposed feasible suggestions.

Local Manifold-Based Sparse Discriminant Learning for Feature Extraction of Hyperspectral Image.

Hyperspectral image (HSI) generally contains a complex manifold structure and strong sparse correlation in its nonlinear high-dimensional data space. However, the existing manifold learning and sparse learning methods usually consider the manifold structure and sparse relationship separately rather than combining manifold and sparse properties to discover the intrinsic information in the original data. To simultaneously reveal the complex sparse relation and manifold structure of HSI, a novel feature extraction (FE) method, called local manifold-based sparse discriminant learning (LMSDL), has been proposed on the basis of manifold learning and sparse representation (SR). The LMSDL method first designs a new sparse optimization model called local manifold-based SR (LMSR) to reveal the local manifold-based sparse structure of data. Then, two geometrical sparse graphs are constructed to represent the discriminant relationship between samples and the geometrical and sparse neighbors. An objective function is constructed via geometrical sparse graphs and reconstruction points to learn a projection matrix for FE. The LMSDL effectively reveals the complex sparse relation and manifold structure in high-dimensional data, and it enhances the representation ability of extracted features for HSI classification significantly. The experimental results on the three real HSI datasets show that the proposed LMSDL algorithm possesses better performance in comparison with some state-of-the-art FE methods.