Curse Of Dimensionality Research Articles

Deep partially linear cox model for current status data.

Deep learning has continuously attained huge success in diverse fields, while its application to survival data analysis remains limited and deserves further exploration. For the analysis of current status data, a deep partially linear Cox model is proposed to circumvent the curse of dimensionality. Modeling flexibility is attained by using deep neural networks (DNNs) to accommodate nonlinear covariate effects and monotone splines to approximate the baseline cumulative hazard function. We establish the convergence rate of the proposed maximum likelihood estimators. Moreover, we derive that the finite-dimensional estimator for treatment covariate effects is $\sqrt{n}$-consistent, asymptotically normal, and attains semiparametric efficiency. Finally, we demonstrate the performance of our procedures through extensive simulation studies and application to real-world data on news popularity.

A Study on Dimensionality Reduction and Parameters for Hyperspectral Imagery Based on Manifold Learning.

With the rapid advancement of remote-sensing technology, the spectral information obtained from hyperspectral remote-sensing imagery has become increasingly rich, facilitating detailed spectral analysis of Earth's surface objects. However, the abundance of spectral information presents certain challenges for data processing, such as the "curse of dimensionality" leading to the "Hughes phenomenon", "strong correlation" due to high resolution, and "nonlinear characteristics" caused by varying surface reflectances. Consequently, dimensionality reduction of hyperspectral data emerges as a critical task. This paper begins by elucidating the principles and processes of hyperspectral image dimensionality reduction based on manifold theory and learning methods, in light of the nonlinear structures and features present in hyperspectral remote-sensing data, and formulates a dimensionality reduction process based on manifold learning. Subsequently, this study explores the capabilities of feature extraction and low-dimensional embedding for hyperspectral imagery using manifold learning approaches, including principal components analysis (PCA), multidimensional scaling (MDS), and linear discriminant analysis (LDA) for linear methods; and isometric mapping (Isomap), locally linear embedding (LLE), Laplacian eigenmaps (LE), Hessian locally linear embedding (HLLE), local tangent space alignment (LTSA), and maximum variance unfolding (MVU) for nonlinear methods, based on the Indian Pines hyperspectral dataset and Pavia University dataset. Furthermore, the paper investigates the optimal neighborhood computation time and overall algorithm runtime for feature extraction in hyperspectral imagery, varying by the choice of neighborhood k and intrinsic dimensionality d values across different manifold learning methods. Based on the outcomes of feature extraction, the study examines the classification experiments of various manifold learning methods, comparing and analyzing the variations in classification accuracy and Kappa coefficient with different selections of neighborhood k and intrinsic dimensionality d values. Building on this, the impact of selecting different bandwidths t for the Gaussian kernel in the LE method and different Lagrange multipliers λ for the MVU method on classification accuracy, given varying choices of neighborhood k and intrinsic dimensionality d, is explored. Through these experiments, the paper investigates the capability and effectiveness of different manifold learning methods in feature extraction and dimensionality reduction within hyperspectral imagery, as influenced by the selection of neighborhood k and intrinsic dimensionality d values, identifying the optimal neighborhood k and intrinsic dimensionality d value for each method. A comparison of classification accuracies reveals that the LTSA method yields superior classification results compared to other manifold learning approaches. The study demonstrates the advantages of manifold learning methods in processing hyperspectral image data, providing an experimental reference for subsequent research on hyperspectral image dimensionality reduction using manifold learning methods.

Interpretable linear dimensionality reduction based on bias-variance analysis

AbstractOne of the central issues of several machine learning applications on real data is the choice of the input features. Ideally, the designer should select a small number of the relevant, nonredundant features to preserve the complete information contained in the original dataset, with little collinearity among features. This procedure helps mitigate problems like overfitting and the curse of dimensionality, which arise when dealing with high-dimensional problems. On the other hand, it is not desirable to simply discard some features, since they may still contain information that can be exploited to improve results. Instead, dimensionality reduction techniques are designed to limit the number of features in a dataset by projecting them into a lower dimensional space, possibly considering all the original features. However, the projected features resulting from the application of dimensionality reduction techniques are usually difficult to interpret. In this paper, we seek to design a principled dimensionality reduction approach that maintains the interpretability of the resulting features. Specifically, we propose a bias-variance analysis for linear models and we leverage these theoretical results to design an algorithm, Linear Correlated Features Aggregation (LinCFA), which aggregates groups of continuous features with their average if their correlation is “sufficiently large”. In this way, all features are considered, the dimensionality is reduced and the interpretability is preserved. Finally, we provide numerical validations of the proposed algorithm both on synthetic datasets to confirm the theoretical results and on real datasets to show some promising applications.

Neural Network Approximation for Pessimistic Offline Reinforcement Learning

Deep reinforcement learning (RL) has shown remarkable success in specific offline decision-making scenarios, yet its theoretical guarantees are still under development. Existing works on offline RL theory primarily emphasize a few trivial settings, such as linear MDP or general function approximation with strong assumptions and independent data, which lack guidance for practical use. The coupling of deep learning and Bellman residuals makes this problem challenging, in addition to the difficulty of data dependence. In this paper, we establish a non-asymptotic estimation error of pessimistic offline RL using general neural network approximation with C-mixing data regarding the structure of networks, the dimension of datasets, and the concentrability of data coverage, under mild assumptions. Our result shows that the estimation error consists of two parts: the first converges to zero at a desired rate on the sample size with partially controllable concentrability, and the second becomes negligible if the residual constraint is tight. This result demonstrates the explicit efficiency of deep adversarial offline RL frameworks. We utilize the empirical process tool for C-mixing sequences and the neural network approximation theory for the Holder class to achieve this. We also develop methods to bound the Bellman estimation error caused by function approximation with empirical Bellman constraint perturbations. Additionally, we present a result that lessens the curse of dimensionality using data with low intrinsic dimensionality and function classes with low complexity. Our estimation provides valuable insights into the development of deep offline RL and guidance for algorithm model design.

On the Approximability and Curse of Dimensionality of Certain Classes of High-Dimensional Functions

On the Approximability and Curse of Dimensionality of Certain Classes of High-Dimensional Functions

A two-stage frequency-domain generation algorithm based on differential evolution for black-box adversarial samples

Adversarial sample generation problem is a hot issue in the security field of deep learning. Evolutionary algorithm has been widely used to solve this problem in recent years because of its good global search ability. However, existing methods still suffer from the “curse of dimensionality” when attacking high-resolution images. In this paper, a two-stage frequency domain generation algorithm of black-box adversarial samples based on differential evolution is proposed. In the first stage, a representative image-guided differential evolution method is proposed to quickly generate a universal adversarial perturbation with a high attack success rate in the frequency-domain. In the second stage, a space reduction strategy based on frequency-domain pixel blocks is designed to reduce the search space and alleviate the problem of “curse of dimensionality”. In addition, a new space–frequency interaction sensitivity measure is introduced to evaluate the similarity between the adversarial samples and the original images. The adversarial perturbations obtained by the measure are more in line with the subjective perception of the human eye. Finally, compared with several typical black-box adversarial sample generation algorithms, experimental results show that the proposed algorithm can achieve higher attack success rate with less prediction times.

Electricity theft detection in smart grid using machine learning

Nowadays, electricity theft is a major issue in many countries and poses a significant financial loss for global power utilities. Conventional Electricity Theft Detection (ETD) models face challenges such as the curse of dimensionality and highly imbalanced electricity consumption data distribution. To overcome these problems, a hybrid system Multi-Layer Perceptron (MLP) approach with Gated Recurrent Units (GRU) is proposed in this work. The proposed hybrid system is applied to analyze and solve electricity theft using data from the Chinese National Grid Corporation (CNGC). In the proposed hybrid system, first, preprocess the data; second, balance the data using the k-means Synthetic Minority Oversampling Technique (SMOTE) technique; third, apply the GTU model to the extracted purified data; fourth, apply the MLP model to the extracted purified data; and finally, evaluate the performance of the proposed system using different performance measures such as graphical analysis and a statistical test. To verify the consistency of our proposed hybrid system, we use three different ratios for training and testing the dataset. The outcomes show that the proposed hybrid system for ETD is highly accurate and efficient compared to the other models like Alexnet, GRU, Bidirectional Gated Recurrent Unit (BGRU) and Recurrent Neural Network (RNN).

Smooth Nested Simulation: Bridging Cubic and Square Root Convergence Rates in High Dimensions

Nested simulation concerns estimating functionals of a conditional expectation via simulation. In this paper, we propose a new method based on kernel ridge regression to exploit the smoothness of the conditional expectation as a function of the multidimensional conditioning variable. Asymptotic analysis shows that the proposed method can effectively alleviate the curse of dimensionality on the convergence rate as the simulation budget increases, provided that the conditional expectation is sufficiently smooth. The smoothness bridges the gap between the cubic root convergence rate (that is, the optimal rate for the standard nested simulation) and the square root convergence rate (that is, the canonical rate for the standard Monte Carlo simulation). We demonstrate the performance of the proposed method via numerical examples from portfolio risk management and input uncertainty quantification. This paper was accepted by Baris Ata, stochastic models and simulation. Funding: The authors acknowledge financial support from the National Natural Science Foundation of China [Grant NSFC 12101149] and the Hong Kong Research Grants Council [Grants GRF 17201520 and GRF 17206821]. Supplemental Material: The e-companion and data files are available at https://doi.org/10.1287/mnsc.2022.00204 .

Benchmarking feature selection and feature extraction methods to improve the performances of machine-learning algorithms for patient classification using metabolomics biomedical data

ObjectiveClassification tasks are an open challenge in the field of biomedicine. While several machine-learning techniques exist to accomplish this objective, several peculiarities associated with biomedical data, especially when it comes to omics measurements, prevent their use or good performance achievements. Omics approaches aim to understand a complex biological system through systematic analysis of its content at the molecular level. On the other hand, omics data are heterogeneous, sparse and affected by the classical “curse of dimensionality” problem, i.e. having much fewer observation, samples (n) than omics features (p). Furthermore, a major problem with multi-omics data is the imbalance either at the class or feature level. The objective of this work is to study whether feature extraction and/or feature selection techniques can improve the performances of classification machine-learning algorithms on omics measurements. Methods:Among all omics, metabolomics has emerged as a powerful tool in cancer research, facilitating a deeper understanding of the complex metabolic landscape associated with tumorigenesis and tumor progression. Thus, we selected three publicly available metabolomics datasets, and we applied several feature extraction techniques both linear and non-linear, coupled or not with feature selection methods, and evaluated the performances regarding patient classification in the different configurations for the three datasets. Results:We provide general workflow and guidelines on when to use those techniques depending on the characteristics of the data available. To further test the extension of our approach to other omics data, we have included a transcriptomics and a proteomics data. Overall, for all datasets, we showed that applying supervised feature selection improves the performances of feature extraction methods for classification purposes. Scripts used to perform all analyses are available at: https://github.com/Plant-Net/Metabolomic_project/.

Tensor networks for solving the time-independent Boltzmann neutron transport equation

Tensor network techniques, known for their low-rank approximation ability that breaks the curse of dimensionality, are emerging as a foundation of new mathematical methods for ultra-fast numerical solutions of high-dimensional Partial Differential Equations (PDEs). Here, we present a mixed Tensor Train (TT)/Quantized Tensor Train (QTT) approach for the numerical solution of time-independent Boltzmann Neutron Transport equations (BNTEs) in Cartesian geometry. Discretizing a realistic three-dimensional (3D) BNTE by (i) diamond differencing in space, (ii) multigroup-in-energy, and (iii) discrete ordinates collocation in angle leads to large generalized eigenvalue problems that generally require a matrix-free approach and large computer clusters. Starting from this discretization, we construct a TT representation of the PDE fields and discrete operators, followed by a QTT representation of the TT cores. We then solve the tensorized generalized eigenvalue problem using a fixed-point scheme with tensor network optimization techniques. We validate our approach by applying the method to two examples of 3D neutron transport problems, currently solved by the Los Alamos National Laboratory PARallel TIme-dependent SN (PARTISN) solver.1 We demonstrate that our TT/QTT method, executed on a standard desktop computer, leads to large compression. This allows for the storage of terrabyte-sized neutron angular flux eigenvectors in megabytes. Additionally, we create megabyte-sized full access TT representations of yottabyte-sized transport matrix operators. By leveraging the TT operators and solution methods, we obtain a 7500 times speedup when compared to the PARTISN solution time with an error of less than 10−5.

An enhanced mechanism for detection of spam emails by deep learning technique with bio-inspired algorithm

This study aims to create a highly effective system for classifying email spam, with the key objective of improving performance and accuracy in classification. Rigorous pre-processing techniques, including lemmatization and tokenization, are applied to refine the dataset. The impact of optimal feature selection on deep learning algorithm stability with varying training datasets is investigated and to improve classifier performance, pre-stages such as feature extraction and selection using bag of words are utilized. The hybrid model is further enhanced by combining feature extraction and classification algorithms with optimization. The inclusion of multiple features, such as n-gram features, addresses the 'Curse of Dimensionality,' and optimal feature selection is employed to improve the spam detection process. The proposed system undergoes three main modifications: multiple feature extraction, feature selection, and an enhanced hybrid model, all aimed at improving spam detection accuracy. The e-mail classification phase involves mapping between training and testing sets, with deep learning algorithms utilized for feature extraction and classification.The recently introduced Sand Cat Swarm Optimization algorithm is employed to refine weights during each epoch, enhancing accuracy and minimizing loss. The proposed system extends its capabilities to identify phishing attacks within the network, offering a comprehensive approach to email security.

Dynamic Exploration–Exploitation Pareto Approach for high-dimensional expensive black-box optimization

Surrogate Optimization (SO) plays a vital role in optimizing performance parameters for computationally expensive simulations. However, SO encounters significant challenges in high-dimensional spaces due to the curse of dimensionality, hampering effective point sampling around global optima. In this paper, we introduce “Dynamic Exploration–Exploitation Pareto Approach (DEEPA),” a novel SO method that combines Pareto sampling with a dynamic discretization schema to optimize high-dimensional black-box functions. Unlike traditional SO methods that heavily rely on specific surrogate models, Pareto sampling offers a more adaptable approach. Improvement-based acquisition functions, frequently employed in black-box optimization, are sensitive to model accuracy and tend to prioritize exploitation, potentially missing valuable regions of interest in complex landscapes. Furthermore, they can encounter challenges when dealing with high-dimensional problems due to the curse of dimensionality. DEEPA leverages dynamic coordinate importance to generate samples effectively, providing a solution for addressing high-dimensionality and complex functions. We employ feature selection strategies to assign importance probabilities to perturb each coordinate, demonstrating the impact of importance-based perturbation on convergence to a near-optimal region. We showcase DEEPA’s versatility in fixed-batch evaluation environments using complex global optimization test problems with various topological properties. We compare DEEPA’s performance with state-of-the-art black-box optimization algorithms, and our experimental results demonstrate DEEPA’s superior performance, particularly in complex problems with multiple local minima.

Compressor airfoil optimization method driven by data-mechanism integration based on evolutionary multi-tasking algorithm

To address the challenge of the "curse of dimensionality" in aerodynamic design optimization of compressors, this study introduces an innovative optimization technique suitable for compressor airfoil design. This technique, rooted in a hybrid mechanism-data-driven approach, seamlessly integrates a hierarchical parameterization method, based on elliptic topological deformation, into a multitasking evolutionary algorithm framework. This integration deviates from the conventional approach of treating parameterization methods and optimization algorithms as distinct elements. The proposed method positions airfoil parameterization as its core, constructing two tasks within the optimization algorithm. It leverages the critical influence of the parameterization method on the aerodynamic performance landscape of the airfoils and the intrinsic qualities of the hierarchical parameterization method in the design space. The multitasking evolutionary optimization framework facilitates effective information exchange between tasks, significantly boosting optimization efficiency. In comparison to standard data-driven multitasking evolutionary algorithms, the proposed method achieves superior optimized solutions with merely 11 × D aerodynamic performance evaluations, where D denotes the number of design variables.

Predicting multiple observations in complex systems through low-dimensional embeddings

Forecasting all components in complex systems is an open and challenging task, possibly due to high dimensionality and undesirable predictors. We bridge this gap by proposing a data-driven and model-free framework, namely, feature-and-reconstructed manifold mapping (FRMM), which is a combination of feature embedding and delay embedding. For a high-dimensional dynamical system, FRMM finds its topologically equivalent manifolds with low dimensions from feature embedding and delay embedding and then sets the low-dimensional feature manifold as a generalized predictor to achieve predictions of all components. The substantial potential of FRMM is shown for both representative models and real-world data involving Indian monsoon, electroencephalogram (EEG) signals, foreign exchange market, and traffic speed in Los Angeles Country. FRMM overcomes the curse of dimensionality and finds a generalized predictor, and thus has potential for applications in many other real-world systems.

Aerodynamic shape optimization based on proper orthogonal decomposition reparameterization under small training sets

Complex shape aerodynamic optimization frequently encounters the problem of dimensionality curse. Extracting geometric features from the original design space proves effective in this issue by reconstructing shapes with lower-dimension. While deep learning methods possess strong capabilities in dimensionality reduction, they suffer from longer training times and higher computational resource requirements. However, Proper Orthogonal Decomposition (POD) presents a more practical algorithm that can be effectively utilized for complex high-dimensional shapes. In this study, we propose a combination of the POD-based method and Gaussian Process Regression with an automatic kernel construction algorithm (AKC-GPR) to optimize complex shapes in a lower-dimensional reparameterization space using small training sets. This approach aims to explore the potential for reducing optimization cost. By reducing the design parameters from 56 to 18 with an error rate of 1.28 %, we establish surrogate models for optimization using AKC-GPR with progressively decreasing training sets. For samples below 100, the Mean Absolute Percentage Errors (MAPEs) remain below 2 %, while the Mean Relative Percentage Errors (MRPEs) approach 10 %. It emphasizes the proficiency of POD in geometric dimensionality reduction. The optimized shapes exhibit smoother contour, resulting in a 10 % improvement in lift and a relatively modest reduction in drag of less than 3 %. The lift-to-drag ratio experiences an increase of over 14 %. The shapes optimized by various training sets possess minimal differences in geometric shapes and aerodynamic characteristics. This confirms the AKC-GPR algorithm's high-precision fitting capability in effectively handling complex aerodynamic coefficients of multi-parameter shapes during small-sample shape optimization.

Mayfly algorithm-based semi-supervised band selection with enhanced bitonic filter for spectral-spatial hyperspectral image classification

ABSTRACT Hyperspectral imaging effectively distinguishes land cover classes with discriminative spectral-spatial characteristics. However, hyperspectral image (HSI) classification encounters three significant challenges: the curse of dimensionality, the scarcity of training samples, and the necessity to incorporate spatial information within the classification process. The large number of redundant and irrelevant spectral bands in HSI can negatively impact the classification performance. Furthermore, the annotation process for training samples is expensive and time-consuming. To address these challenges, this work presents a novel semi-supervised approach for hyperspectral image dimensionality reduction using the Mayfly Algorithm (MA). The proposed method aims to preserve the most informative spectral bands while leveraging both labelled and unlabelled samples in the process. The MA algorithm effectively balances exploration and exploitation strategies, enhancing population diversity and mitigating challenges such as premature convergence and local optima. Additionally, the inclusion of structural variation and thresholding (SVT) in the Bitonic filter enables the incorporation of spatial features during the classification phase. This improves the separability of pixels belonging to different objects and leads to significant improvements in classification performance. Finally, two machine learning classifiers, namely Random Forest (RF) and Support Vector Machine (SVM), are applied at the pixel level for hyperspectral image classification. Extensive experimental results on three real hyperspectral datasets demonstrate the superiority of the proposed approach compared to state-of-the-art algorithms like Harmony Search (HS), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and more with an average improvement of 3.2%.

Sophisticated and small versus simple and sizeable: When does it pay off to introduce drifting coefficients in Bayesian vector autoregressions?

AbstractWe assess the relationship between model size and complexity in the time‐varying parameter vector autoregression (VAR) framework via thorough predictive exercises for the euro area, the United Kingdom, and the United States. It turns out that sophisticated dynamics through drifting coefficients are important in small data sets, while simpler models tend to perform better in sizeable data sets. To combine the best of both worlds, novel shrinkage priors help to mitigate the curse of dimensionality, resulting in competitive forecasts for all scenarios considered. Furthermore, we discuss dynamic model selection to improve upon the best performing individual model for each point in time.

An improved loci method for outlier detection in fuzzy datasets based on fractional distance metric and outlierness degree

Density based methods are significant approaches in outlier detection for high dimensional datasets and Local correlation integral (LOCI) is one of the best of them. To extend LOCI for fuzzy datasets, we should employ suitable metrics to measure the distance between two fuzzy numbers. Euclidean distance measure is a classic one in metric learning, but to overcome curse of dimensionality, we apply fractional distance metric too. Then, after introducing the FLOCI outlier detection algorithm for identifying the fuzzy outliers, we study the efficiency of the proposed method by doing some numerical experiments, in which the obtained results were completely successfull. We also compared the results with Fuzzy versions of Distance based ABOD and SOD methods to prove robustness of this approache. More than the above, one of the main advantages of the new approach is the determination of outlierness factor for each data which is not presented in classical LOCI method.

Approximation of optimization problems with constraints through kernel Sum-of-Squares

Handling an infinite number of inequality constraints in infinite-dimensional spaces occurs in many fields, from global optimization to optimal transport. These problems have been tackled individually in several previous articles through kernel Sum-of-Squares (kSoS) approximations. We propose here a unified theorem to prove convergence guarantees for these schemes. Pointwise inequalities are turned into equalities within a class of nonnegative kSoS functions. Assuming further that the functions appearing in the problem are smooth, focusing on pointwise equality constraints enables the use of scattering inequalities to mitigate the curse of dimensionality in sampling the constraints. Our approach is illustrated in learning vector fields with side information, here the invariance of a set.

Multi-label feature selection via latent representation learning and dynamic graph constraints

As an effective method to deal with the curse of dimensionality, multi-label feature selection aims to select the most representative subset of features by eliminating unfavorable features. Although great progress has been made in this field, how to mine adequate supervisory information from multi-label data remains a key challenge. Compared to the latent information of instances, the latent information of instance relevance contains both the basic information of instances and the latent relevance between instances. Base on this knowledge, we propose a novel multi-label feature selection method named LRDG that explores latent representation learning and dynamic graph constraints. Specifically, we introduce the latent representation of instance relevance as supervisory information for pseudo-label learning, and minimize information loss during pseudo-label learning by means of the label manifold, the non-negative constraints, and the minimization of the Frobenius norm between pseudo-labels and ground-truth labels. In addition, considering the shortcomings brought by traditional graph regularization, we propose to use the dynamic graph constructed from low-dimensional pseudo-labels to constrain feature weights. Extensive experiments on various multi-label datasets demonstrate the effectiveness of the proposed method. The code is available at https://github.com/yunbao520/LRDG.