Problem Of Dimensionality Reduction Research Articles

Minimax rates of convergence for sliced inverse regression with differential privacy

Sliced inverse regression (SIR) is a highly efficient paradigm used for the purpose of dimension reduction by replacing high-dimensional covariates with a limited number of linear combinations. This paper focuses on the implementation of the classical SIR approach integrated with a Gaussian differential privacy mechanism to estimate the central space while preserving privacy. We illustrate the tradeoff between statistical accuracy and privacy in sufficient dimension reduction problems under both the classical low- dimensional and modern high-dimensional settings. Additionally, we achieve the minimax rate of the proposed estimator with Gaussian differential privacy constraint and illustrate that this rate is also optimal for multiple index models with bounded dimension of the central space. Extensive numerical studies on synthetic data sets are conducted to assess the effectiveness of the proposed technique in finite sample scenarios, and a real data analysis is presented to showcase its practical application.

Conditional mean dimension reduction for tensor time series

The dimension reduction problem for a stationary tensor time series is addressed. The goal is to remove linear combinations of the tensor time series that are mean independent of the past, without imposing any parametric models or distributional assumptions. To achieve this goal, a new metric called cumulative tensor martingale difference divergence is introduced and its theoretical properties are studied. Unlike existing methods, the proposed approach achieves dimension reduction by estimating a distinctive subspace that can fully retain the conditional mean information. By focusing on the conditional mean, the proposed dimension reduction method is potentially more accurate in prediction. The method can be viewed as a factor model-based approach that extends the existing techniques for estimating central subspace or central mean subspace in vector time series. The effectiveness of the proposed method is illustrated by extensive simulations and two real-world data applications.

Gradient‐based approach to sufficient dimension reduction with functional or longitudinal covariates

AbstractIn this paper, we focus on the sufficient dimension reduction problem in regression analysis with real‐valued response and functional or longitudinal covariates. We propose a new method based on gradients of the conditional distribution function to estimate the sufficient dimension reduction subspace. While existing inverse‐regression‐type methods relies on a linearity condition, our method is based on the gradient of conditional distribution function and its validity only requires smoothness conditions on the population parameters. Practically, the proposed estimator can be obtained by standard algorithm of functional principal component analysis. The proposed method is demonstrated through extensive simulations and two empirical examples.

The Alternating Direction Method of Multipliers for Sufficient Dimension Reduction

The minimum average variance estimation (MAVE) method has proven to be an effective approach to sufficient dimension reduction. In this study, we apply the computationally efficient optimization algorithm named alternating direction method of multipliers (ADMM) to a particular approach (MAVE or minimum average variance estimation) to the problem of sufficient dimension reduction (SDR). Under some assumptions, we prove that the iterative sequence generated by ADMM converges to some point of the associated augmented Lagrangian function. Moreover, that point is stationary. It also presents some numerical simulations on synthetic data to demonstrate the computational efficiency of the algorithm.

Gauge-Optimal Approximate Learning for Small Data Classification.

Small data learning problems are characterized by a significant discrepancy between the limited number of response variable observations and the large feature space dimension. In this setting, the common learning tools struggle to identify the features important for the classification task from those that bear no relevant information and cannot derive an appropriate learning rule that allows discriminating among different classes. As a potential solution to this problem, here we exploit the idea of reducing and rotating the feature space in a lower-dimensional gauge and propose the gauge-optimal approximate learning (GOAL) algorithm, which provides an analytically tractable joint solution to the dimension reduction, feature segmentation, and classification problems for small data learning problems. We prove that the optimal solution of the GOAL algorithm consists in piecewise-linear functions in the Euclidean space and that it can be approximated through a monotonically convergent algorithm that presents-under the assumption of a discrete segmentation of the feature space-a closed-form solution for each optimization substep and an overall linear iteration cost scaling. The GOAL algorithm has been compared to other state-of-the-art machine learning tools on both synthetic data and challenging real-world applications from climate science and bioinformatics (i.e., prediction of the El Niño Southern Oscillation and inference of epigenetically induced gene-activity networks from limited experimental data). The experimental results show that the proposed algorithm outperforms the reported best competitors for these problems in both learning performance and computational cost.

Dimensionality Reduction problem: a Comprehensive Exploration of Disjoint Principal Component Analysis (DPCA) and Disjoint Multiple Correspondence Analysis (DMCA)

This paper delves into the realm of advanced data analysis, focusing on two powerful dimensionality reduction methods: Disjoint Principal Component Analysis (DPCA) and Disjoint Multiple Correspondence Analysis (DMCA). Methodological marvels in their own right, these approaches are scrutinized for their unique properties and applications across diverse domains. We navigate through the intricacies of their algorithms and explore how they unveil patterns within complex datasets. The comparative analysis highlights the strengths and weaknesses of DPCA and DMCA, shedding light on their distinct contributions to the analytical landscape. This paper serves as a comprehensive guide for researchers and analysts seeking deeper insights into these cutting-edge techniques for dimensional reduction.

Attribute reduction with fuzzy kernel-induced relations

Fuzzy rough sets have strong ability in dealing with data uncertainty, which have been widely applied in attribute reduction problems. Fuzzy relations are the cornerstones of fuzzy rough sets, and kernel-induced relations can simultaneously reflect the similarities of objects in both original space and high-dimensional Hilbert space. However, existing distance-based fuzzy relations (even kernel-induced) cannot truly reflect the correlations of objects under complex data distributions. To obtain better expressing ability for data, we propose a general optimization framework to get the memberships of objects with respect to categories. Considering various complex data distributions, the concepts of intra-class aggregation degree based on membership and inter-class dispersion degree based on nonmembership are proposed, and a separability measure is constructed to evaluate the importance of attributes to decisions. Correspondingly, a new attribute reduction algorithm is proposed, with a forward evaluation L-step strategy to skip local fluctuations in classification accuracy. Finally, we conduct a series of experiments to verify the effectiveness of the proposed algorithm. From the experiments on UCI, ELVIRA and face recognition datasets, it can be seen that the attribute subset obtained by the proposed algorithm not only has high classification accuracy and computational efficiency, but also has fewer attributes.

Causality-Based PCA Methods for Condition Modeling of Mechatronic Systems

This paper investigates the feature dimensionality reduction problem of high-dimensional data in condition modelling of complex mechatronic systems, aiming at improving performance, robustness and interpretability of classical principle component analysis (PCA) methods. First, the causal discovery algorithms are adopted to construct the corresponding causal network among monitoring variables. Then, two causality-based PCA methods are proposed, based on the corresponding qualitative reachability matrix and quantitative reachability strength matrix, respectively. Average treatment effect (ATE) values are considered as the quantitative measure of the causalities in a reachability strength matrix. The robustness and effectiveness of the proposed methods are verified on a synthetic dataset and four public datasets in comparison with two benchmark methods, i.e. PCA and autoencoder (AE). The proposed methods are also adopted in a high-speed train braking system for extracting informative features from the monitoring variables.

Dimensionality Reduction for Hierarchical Multi-Label Classification: A Systematic Mapping Study

Hierarchical multi-label classification problems typically deal with datasets with many attributes and labels, which can negatively impact the classifier performance. The application of dimensionality reduction methods can significantly improve the performance of classifiers. Dimensionality reduction can be performed by feature extraction or feature selection, according to the problem domain and datasets characteristics. This work carried out a systematic literature mapping to identify the approaches and techniques of dimensionality reduction that have been used in hierarchical multi-label classification tasks. Searches were performed on 7 important databases for the Computer Science field. From a list of 184 retrieved papers, 12 were selected for analysis, from which it was possible to determine a general overview of studies conducted from 2010 to 2022. It was identified that feature selection was the most frequent reduction method, with filter approach standing out. In addition, it was detected that most of the works used tree hierarchical structure. As its main outcome, this paper presents the state of the art of dimensionality reduction problem for hierarchical multi-label classification, indicating trends and research issues in the field.

Deep Ensemble Transformers for Dimensionality Reduction.

We propose deep ensemble transformers (DETs), a fast, scalable approach for dimensionality reduction problems. This method leverages the power of deep neural networks and employs cascade ensemble techniques as its fundamental feature extraction tool. To handle high-dimensional data, our approach employs a flexible number of intermediate layers sequentially. These layers progressively transform the input data into decision tree predictions. To further enhance prediction performance, the output from the final intermediate layer is fed through a feed-forward neural network architecture for final prediction. We derive an upper bound of the disparity between the generalization error and the empirical error and demonstrate that it converges to zero. This highlights the generalizability of our method to parameter estimation and feature selection problems. In our experimental evaluations, DETs outperform existing models in terms of prediction accuracy, representation learning ability, and computational time. Specifically, the method achieves over 95% accuracy in gene expression data and can be trained on average 50% faster than traditional artificial neural networks (ANNs).

Likelihood-based surrogate dimension reduction

We consider the problem of surrogate sufficient dimension reduction, that is, estimating the central subspace of a regression model, when the covariates are contaminated by measurement error. When no measurement error is present, a likelihood-based dimension reduction method that relies on maximizing the likelihood of a Gaussian inverse regression model on the Grassmann manifold is well-known to have superior performance to traditional inverse moment methods. We propose two likelihood-based estimators for the central subspace in measurement error settings, which make different adjustments to the observed surrogates. Both estimators are computed based on maximizing objective functions on the Grassmann manifold and are shown to consistently recover the true central subspace. When the central subspace is assumed to depend on only a few covariates, we further propose to augment the likelihood function with a penalty term that induces sparsity on the Grassmann manifold to obtain sparse estimators. The resulting objective function has a closed-form Riemann gradient which facilitates efficient computation of the penalized estimator. We leverage the state-of-the-art trust region algorithm on the Grassmann manifold to compute the proposed estimators efficiently. Simulation studies and a data application demonstrate the proposed likelihood-based estimators perform better than inverse moment-based estimators in terms of both estimation and variable selection accuracy.

Dimensionality reduction in diffusion–reaction systems

AbstractThe generalized solutions of the dimensionality reduction problem in diffusion–reaction systems have been found for the first time by rigorously decoupling probability functions. Numerically exact results are obtained by straightforwardly applying the generalized solution to three models: protein‐DNA binding model (3D‐1D reduction), surface‐mediated diffusion (3D‐2D reduction), and line‐mediated diffusion cases (2D‐1D reduction). The results are confirmed by Monte Carlo simulations.

Sparse robust adaptive unsupervised subspace learning for dimensionality reduction

This work is devoted to the investigation of dimension reduction problem. As an efficient dimension reduction method, much attention has been paid on unsupervised subspace learning since it does not rely on expensive labels. Firstly, we implant a robust estimator in the error term of objective function, this leads to that small coefficients can be automatically allocated to the abnormal points. Thus, our model is robust to noise and outliers. Posteriorly, the L2,r-norm (1≤r≤2) is used as a measure of error, then, the performance of the model can be improved by selecting the appropriate adaptive parameter r. Further, a L2,p-norm (0<p≤1) regularization term is added to the objective function, therefore the gained sparse subspace can further improve the efficiency and accuracy of the expression, as well as enhance the generalization ability of the model and reduce over-fitting. Moreover, an efficient algorithm with fast convergence speed is designed to solve the model. Finally, the experimental results on 14 datasets show that the subspace dimension obtained by our method is no more than 6. From the results of classification test, our algorithm has obvious advantages over the other similar six algorithms.

Data‐driven state observation for nonlinear systems based on online learning

AbstractFor controlling nonlinear processes represented by state‐space models, a state observer is needed to estimate the states from the trajectories of measured variables. While model‐based observer synthesis is traditionally challenging due to the difficulty of solving pertinent partial differential equations, this article proposes an efficient model‐free, data‐driven approach for state observation, which is suitable for data‐driven nonlinear control without accurate nonlinear models. Specifically, by using a Chen–Fliess series representation of the observer dynamics, state observation is endowed with an online least squares regression formulation that can be solved by gradient flow with performance guarantees. When the target state trajectories for regression are unavailable, by exploiting the Kazantzis–Kravaris/Luenberger observer structure, state observation is reduced to a dimensionality reduction problem amenable to an online implementation of kernel principal component analysis. The proposed approach is demonstrated by a limit cycle dynamics and a chaotic system.

Supervised Dimensionality Reduction of Proportional Data Using Exponential Family Distributions

Most well-known supervised dimensionality reduction algorithms suffer from the curse of dimensionality while handling high-dimensional sparse data due to ill-conditioned second-order statistics matrices. They also do not deal with multi-modal data properly since they construct neighborhood graphs that do not discriminate between multi-modal classes of data and single-modal ones. In this paper, a novel method that mitigates the above problems is proposed. In this method, assuming the data is from two classes, they are projected into the low-dimensional space in the first step which removes sparsity from the data and reduces the time complexity of any operation drastically afterwards. These projected data are modeled using a mixture of exponential family distributions for each class, allowing the modeling of multi-modal data. A measure for the similarity between the two projected classes is used as an objective function for constructing an optimization problem, which is then solved using a heuristic search algorithm to find the best separating projection. The conducted experiments show that the proposed method outperforms the rest of the compared algorithms and provides a robust effective solution to the problem of dimensionality reduction even in the presence of multi-modal and sparse data.

MONITORING DATA AGGREGATION OF DYNAMIC SYSTEMS USING INFORMATION TECHNOLOGIES

The subject matter of the article is models, methods and information technologies of monitoring data aggregation. The goal of the article is to determine the best deep learning model for reducing the dimensionality of dynamic systems monitoring data. The following tasks were solved: analysis of existing dimensionality reduction approaches, description of the general architecture of vanilla and variational autoencoders, development of their architecture, development of software for training and testing of autoencoders, conducting research on the performance quality of autoencoders for the problem of dimensionality reduction. The following models and methods were used: data processing and preparation, data dimensionality reduction. The software was developed using the Python language. Scikit-learn, Pandas, PyTorch, NumPy, argparse and others were used as auxiliary libraries. Obtained results: the work presents a classification of models and methods for dimensionality reduction, general reviews of vanilla and variational autoencoders, which include a description of the models, their properties, loss functions and their application to the problem of dimensionality reduction. Custom autoencoder architectures were also created, including visual representations of the autoencoder architecture and descriptions of each component. The software for training and testing autoencoders was developed, the dynamic system monitoring data set, and the steps for pre-training the data set were described. The metric for evaluating the quality of models is also described; the configuration of autoencoders and their training are considered. Conclusions: The vanilla autoencoder recovers the data much better than the variational one. Looking at the fact that the architectures of the autoencoders are the same, except for the peculiarities of the autoencoders, it can be noted that a vanilla autoencoder compresses data better by keeping more useful variables for later recovery from the bottleneck. Additionally, by training on different bottleneck sizes, you can determine the size at which the data is recovered best, which means that the most important variables are preserved. Looking at the results in general, the autoencoders work effectively for the dimensionality reduction task and the data recovery quality metric shows that they recover the data well with an error of 3–4 digits after 0. In conclusion, the vanilla autoencoder is the best deep learning model for aggregating monitoring data of dynamic systems.

A hybrid framework for short term load forecasting with a navel feature engineering and adaptive grasshopper optimization in smart grid

Short-term load forecasting (STLF) enables distribution system operators to perform efficient energy management by flexibly engaging energy consumers under the intelligent demand-response program in the smart grid (SG). This paper develops a fast and accurate hybrid load forecasting model with this motivation. The proposed model integrates a locally weighted support vector regression (LWSVR) based forecaster with two modules. These modules are feature engineering (FE) and adaptive grasshopper optimization (AGO) based optimizers. In the FE module, firstly, the proposed hybrid feature selector (HFS) is developed using wrapper and filter techniques to determine an optimal subset of features. Furthermore, the instance-based Relief-F (REF) and information theoretic-based mutual information (MI) filters are used to decrease the curse of feature dimensionality by finding and eradicating extraneous features. In addition, to overcome the overfitting problem, the HFS module is further optimized using the recursive feature elimination (RFE) wrapper feature selection technique. The essential features are then extracted using a radial basis Kernel-based principal component analysis (RBF-KPCA) algorithm to eliminate the dimensionality reduction problem. The AGO algorithm tunes the LWSVR model’s appropriate parameters to effectively evade entrapping into local optimum and yield accurate prediction results. However, the efficacy and productiveness of the forecasting model are differentiated equally by its convergence rate and stability. Actual hourly load data of two states of Australia (New South Wales (NSW) and Victoria (VIC)) and California Independent System Operator United States (CAISO-US) are employed as a case study to estimate the effectiveness and applicability of the designed model. Empirical results show that the devised model surpasses benchmark models (single and hybrid) in terms of stability, accuracy, and convergence rate.

Topological reduction approaches for relation decision systems

In this study, we use topological methods to investigate attribute reduction problems for relation decision systems. We propose the notions of topological consistent and inconsistent relation decision systems. Then, we propose the concept of the consistent topological reduction of topological consistent relation decision systems. To enlarge the scope of application, we define the general topological reduction of relation decision systems, which is a further generalization of consistent topological reduction. Moreover, we develop corresponding reduction algorithms for these two types of reductions. To demonstrate that our algorithms cannot be unified by the general reduction algorithm, we discuss the relationship between the consistent topological reducts and the reducts identified by the general reduction algorithm. We conduct numerical experiments on 11 UCI datasets to verify our theoretical results. The experimental results demonstrate that the proposed reduction algorithms are effective and practicable.

Discussion of “Multivariate dynamic modeling for Bayesian forecasting of business revenue”

Discussion of “Multivariate dynamic modeling for Bayesian forecasting of business revenue”

Semi-Supervised Multi-Label Dimensionality Reduction Learning by Instance and Label Correlations

The label learning mechanism is challenging to integrate into the training model of the multi-label feature space dimensionality reduction problem, making the current multi-label dimensionality reduction methods primarily supervision modes. Many methods only focus attention on label correlations and ignore the instance interrelations between the original feature space and low dimensional space. Additionally, very few techniques consider how to constrain the projection matrix to identify specific and common features in the feature space. In this paper, we propose a new approach of semi-supervised multi-label dimensionality reduction learning by instance and label correlations (SMDR-IC, in short). Firstly, we reformulate MDDM which incorporates label correlations as a least-squares problem so that the label propagation mechanism can be effectively embedded into the model. Secondly, we investigate instance correlations using the k-nearest neighbor technique, and then present the l1-norm and l2,1-norm regularization terms to identify the specific and common features of the feature space. Experiments on the massive public multi-label data sets show that SMDR-IC has better performance than other related multi-label dimensionality reduction methods.