Adaptive Dimensionality Reduction Research Articles

Impact of dimensionality reduction techniques on student performance prediction using machine learning

This study addresses the crucial issue of predicting student performance in educational data mining (EDM) by proposing an Adaptive Dimensionality Reduction Algorithm (ADRA). ADRA efficiently reduces the dimensionality of student data, encompassing various academic, demographic, behavioral, social, and health-related features. It achieves this by iteratively selecting the most relevant features based on a combined normalized mean rank of five feature ranking methods. This reduction in dimensionality enhances the performance of predictive models and provides valuable insights into the key factors influencing student performance. The study evaluates ADRA using four different student performance datasets and six machine learning algorithms, comparing it to three existing dimensionality reduction methods. The results show that ADRA achieves an average dimensionality reduction factor of 6.2 while maintaing comprable accuracy with other mehtods.

Low-rank Representation with Adaptive Dimensionality Reduction via Manifold Optimization for Clustering

The dimensionality reduction techniques are often used to reduce data dimensionality for computational efficiency or other purposes in existing low-rank representation (LRR)-based methods. However, the two steps of dimensionality reduction and learning low-rank representation coefficients are implemented in an independent way; thus, the adaptability of representation coefficients to the original data space may not be guaranteed. This article proposes a novel model, i.e., low-rank representation with adaptive dimensionality reduction (LRRARD) via manifold optimization for clustering, where dimensionality reduction and learning low-rank representation coefficients are integrated into a unified framework. This model introduces a low-dimensional projection matrix to find the projection that best fits the original data space. And the low-dimensional projection matrix and the low-rank representation coefficients interact with each other to simultaneously obtain the best projection matrix and representation coefficients. In addition, a manifold optimization method is employed to obtain the optimal projection matrix, which is an unconstrained optimization method in a constrained search space. The experimental results on several real datasets demonstrate the superiority of our proposed method.

Probabilistic partition of unity networks for high‐dimensional regression problems

AbstractWe explore the probabilistic partition of unity network (PPOU‐Net) model in the context of high‐dimensional regression problems and propose a general framework focusing on adaptive dimensionality reduction. With the proposed framework, the target function is approximated by a mixture of experts model on a low‐dimensional manifold, where each cluster is associated with a fixed‐degree polynomial. We present a training strategy that leverages the expectation maximization (EM) algorithm. During the training, we alternate between (i) applying gradient descent to update the DNN coefficients; and (ii) using closed‐form formulae derived from the EM algorithm to update the mixture of experts model parameters. Under the probabilistic formulation, step (ii) admits the form of embarrassingly paralleliazable weighted least‐squares solves. The PPOU‐Nets consistently outperform the baseline fully‐connected neural networks of comparable sizes in numerical experiments of various data dimensions. We also explore the proposed model in applications of quantum computing, where the PPOU‐Nets act as surrogate models for cost landscapes associated with variational quantum circuits.

Enhancing Grid-Based 3D Object Detection in Autonomous Driving With Improved Dimensionality Reduction

Point cloud object detection is a pivotal technology in autonomous driving and robotics. Currently, the majority of cutting-edge point cloud detectors utilize Bird’s Eye View (BEV) for detection, as it allows them to take advantage of well-explored 2D detection techniques. Nevertheless, dimensionality reduction of features from 3D space to BEV space unavoidably leads to information loss, and there is a lack of research on this issue. Existing methods typically obtain BEV features by collapsing voxel or point features along the height dimension via a pooling operation or convolution, resulting in a significant decrease in geometric information. To tackle this problem, we present a new point cloud backbone network for grid-based object detection, MDRNet, which is based on adaptive dimensionality reduction and multi-level spatial residual strategies. In MDRNet, the Spatial-aware Dimensionality Reduction (SDR) is designed to dynamically concentrate on the essential components of the object during 3D-to-BEV transformation. Moreover, the Multi-level Spatial Residuals (MSR) strategy is proposed to effectively fuse multi-level spatial information in BEV feature maps. Our MDRNet can be employed on any existing grid-based object detector, resulting in a remarkable improvement in performance. Numerous experiments conducted on nuScenes, KITTI and DAIR-V have shown that MDRNet surpasses existing SOTA approaches. In particular, on the nuScenes dataset, we attained an impressive 7.2% mAP and 5.0% NDS enhancement compared with CenterPoint.

Multi-user motion recognition using sEMG via discriminative canonical correlation analysis and adaptive dimensionality reduction

The inability of new users to adapt quickly to the surface electromyography (sEMG) interface has greatly hindered the development of sEMG in the field of rehabilitation. This is due mainly to the large differences in sEMG signals produced by muscles when different people perform the same motion. To address this issue, a multi-user sEMG framework is proposed, using discriminative canonical correlation analysis and adaptive dimensionality reduction (ADR). The interface projects the feature sets for training users and new users into a low-dimensional uniform style space, overcoming the problem of individual differences in sEMG. The ADR method removes the redundant information in sEMG features and improves the accuracy of system motion recognition. The presented framework was validated on eight subjects with intact limbs, with an average recognition accuracy of 92.23% in 12 categories of upper-limb movements. In rehabilitation laboratory experiments, the average recognition rate reached 90.52%. The experimental results suggest that the framework offers a good solution to enable new rehabilitation users to adapt quickly to the sEMG interface.

Dynamic Orthogonal Projection Constrained Discriminative Tracking

Due to the end-to-end feature learning with convolutional neural networks (CNNs), modern discriminative trackers improve the state of the art significantly. To achieve a strong discrimination, the learned features are usually high-dimensional, resulting in a massive number of parameters contained in the discriminative model and the increase of risk of over-fitting in the online tracking. In this letter, we try to alleviate the risk of over-fitting by means of the adaptive dimensionality reduction (DR) through CNNs. Specifically, an orthogonality constrained ridge regression model is proposed to reduce the dimensionality of features, and a dynamic sub-network (DOPNet) is designed to learn to perform DR. After trained with an orthogonality loss and a regression one, DOPNet generates a set of orthogonal bases (i. e., weights in FC layers) dynamically to reduce the feature dimensionality for a discriminative model in the online tracking. Based on the novel discriminative model and DOPNet, an effective and efficient tracker, DOPTracker, is developed. DOPTracker achieves the state-of-the-art results on four benchmarks, OTB-2015, VOT-2018, NfS, and GOT-10 k while running at 30 FPS.

Adaptive dimensionality reduction for neural network-based online principal component analysis.

“Principal Component Analysis” (PCA) is an established linear technique for dimensionality reduction. It performs an orthonormal transformation to replace possibly correlated variables with a smaller set of linearly independent variables, the so-called principal components, which capture a large portion of the data variance. The problem of finding the optimal number of principal components has been widely studied for offline PCA. However, when working with streaming data, the optimal number changes continuously. This requires to update both the principal components and the dimensionality in every timestep. While the continuous update of the principal components is widely studied, the available algorithms for dimensionality adjustment are limited to an increment of one in neural network-based and incremental PCA. Therefore, existing approaches cannot account for abrupt changes in the presented data. The contribution of this work is to enable in neural network-based PCA the continuous dimensionality adjustment by an arbitrary number without the necessity to learn all principal components. A novel algorithm is presented that utilizes several PCA characteristics to adaptivly update the optimal number of principal components for neural network-based PCA. A precise estimation of the required dimensionality reduces the computational effort while ensuring that the desired amount of variance is kept. The computational complexity of the proposed algorithm is investigated and it is benchmarked in an experimental study against other neural network-based and incremental PCA approaches where it produces highly competitive results.

A Genetic Optimization Algorithm Based on Adaptive Dimensionality Reduction

With the rise of big data in cloud computing, many optimization problems have gradually developed into high-dimensional large-scale optimization problems. In order to address the problem of dimensionality in optimization for genetic algorithms, an adaptive dimensionality reduction genetic optimization algorithm (ADRGA) is proposed. An adaptive vector angle factor is introduced in the algorithm. When the angle of an individual’s adjacent dimension is less than the angle factor, the value of the smaller dimension is marked as 0. Then, the angle between each individual dimension is calculated separately, and the number of zeros in the population is updated. When the number of zeros of all individuals in a population exceeds a given constant in a certain dimension, the dimension is considered to have no more information and deleted. Eight high-dimensional test functions are used to verify the proposed adaptive dimensionality reduction genetic optimization algorithm. The experimental results show that the convergence, accuracy, and speed of the proposed algorithm are better than those of the standard genetic algorithm (GA), the hybrid genetic and simulated annealing algorithm (HGSA), and the adaptive genetic algorithm (AGA).

Sparse-Adaptive Hypergraph Discriminant Analysis for Hyperspectral Image Classification

Hyperspectral image (HSI) contains complex multiple structures. Therefore, the key problem analyzing the intrinsic properties of an HSI is how to represent the structure relationships of the HSI effectively. Hypergraph is very effective to describe the intrinsic relationships of the HSI. In general, Euclidean distance is adopted to construct the hypergraph. However, this method cannot effectively represent the structure properties of high-dimensional data. To address this problem, we propose a sparse-adaptive hypergraph discriminant analysis (SAHDA) method to obtain the embedding features of the HSI in this letter. SAHDA uses the sparse representation to reveal the structure relationships of the HSI adaptively. Then, an adaptive hypergraph is constructed by using the intraclass sparse coefficients. Finally, we develop an adaptive dimensionality reduction mode to calculate the weights of the hyperedges and the projection matrix. SAHDA can adaptively reveal the intrinsic properties of the HSI and enhance the performance of the embedding features. Some experiments on the Washington DC Mall hyperspectral data set demonstrate the effectiveness of the proposed SAHDA method, and SAHDA achieves better classification accuracies than the traditional graph learning methods.

Hyperspectral imagery classification based on semi-supervised 3-D deep neural network and adaptive band selection

This paper proposes a novel approach based on adaptive dimensionality reduction (ADR) and a semi-supervised 3-D convolutional neural network (3-D CNN) for the spectro-spatial classification of hyperspectral images (HSIs). It tackles the problem of curse of dimensionality and the limited number of training samples by selecting the most relevant spectral bands. The selected bands should be informative, discriminative and distinctive. They are fed into a semi-supervised 3-D CNN feature extractor, then a linear regression classifier to produce the classification map. In fact, the proposed semi-supervised 3-D CNN model seeks to extract the deep spectral and spatial features based on convolutional encoder-decoder to enhance the HSI classification. It uses several 3-D convolution and max-pooling layers to extract these features from the selected relevant bands. The main advantage of the proposed approach is to reduce the high dimensionality of HSI, preserve the relevant spectro-spatial information and enhance the classification using few labeled training samples. Experimental studies are carried out on three real HSI data sets: Indian Pines, Pavia University, and Salinas. The obtained results show that the proposed approach performs better than other deep learning-based methods including CNN-based methods, and significantly improves the classification accuracy of HSIs.

Adaptive Dimensionality Reduction for Fast Sequential Optimization With Gaussian Processes

Available computational models for many engineering design applications are both expensive and and of a black-box nature. This renders traditional optimization techniques difficult to apply, including gradient-based optimization and expensive heuristic approaches. For such situations, Bayesian global optimization approaches, that both explore and exploit a true function while building a metamodel of it, are applied. These methods often rely on a set of alternative candidate designs over which a querying policy is designed to search. For even modestly high-dimensional problems, such an alternative set approach can be computationally intractable, due to the reliance on excessive exploration of the design space. To overcome this, we have developed a framework for the optimization of expensive black-box models, which is based on active subspace exploitation and a two-step knowledge gradient policy. We demonstrate our approach on three benchmark problems and a practical aerostructural wing design problem, where our method performs well against traditional direct application of Bayesian global optimization techniques.

Robust Visual Tracking Based on Adaptive Convolutional Features and Offline Siamese Tracker.

Robust and accurate visual tracking is one of the most challenging computer vision problems. Due to the inherent lack of training data, a robust approach for constructing a target appearance model is crucial. The existing spatially regularized discriminative correlation filter (SRDCF) method learns partial-target information or background information when experiencing rotation, out of view, and heavy occlusion. In order to reduce the computational complexity by creating a novel method to enhance tracking ability, we first introduce an adaptive dimensionality reduction technique to extract the features from the image, based on pre-trained VGG-Net. We then propose an adaptive model update to assign weights during an update procedure depending on the peak-to-sidelobe ratio. Finally, we combine the online SRDCF-based tracker with the offline Siamese tracker to accomplish long term tracking. Experimental results demonstrate that the proposed tracker has satisfactory performance in a wide range of challenging tracking scenarios.

Adaptive Dimensionality-Reduction for Time-Stepping in Differential and Partial Differential Equations

AbstractA numerical time-stepping algorithm for differential or partial differential equations is proposed that adaptively modifies the dimensionality of the underlying modal basis expansion. Specifically, the method takes advantage of any underlying low-dimensional manifolds or subspaces in the system by using dimensionality-reduction techniques, such as the proper orthogonal decomposition, in order to adaptively represent the solution in the optimal basis modes. The method can provide significant computational savings for systems where low-dimensional manifolds are present since the reduction can lower the dimensionality of the underlying high-dimensional system by orders of magnitude. A comparison of the computational efficiency and error for this method are given showing the algorithm to be potentially of great value for high-dimensional dynamical systems simulations, especially where slow-manifold dynamics are known to arise. The method is envisioned to automatically take advantage of any potential computational saving associated with dimensionality-reduction, much as adaptive time-steppers automatically take advantage of large step sizes whenever possible.

Exploring Locally Adaptive Dimensionality Reduction for Hyperspectral Image Classification: A Maximum Margin Metric Learning Aspect

The high-dimensional data space generated by hyperspectral sensors introduces challenges for the conventional data analysis techniques. Popular dimensionality reduction techniques usually assume a Gaussian distribution, which may not be in accordance with real life. Metric learning methods, which explore the global data structure of the labeled training samples, have proved to be very efficient in hyperspectral fields. However, we can go further by utilizing locally adaptive decision constraints for the labeled training samples per class to obtain an even better performance. In this paper, we present the locally adaptive dimensionality reduction metric learning (LADRml) method for hyperspectral image classification. The aims of the presented method are: 1) first, to utilize the limited training samples to reduce the dimensionality of data without a certain distribution hypothesis; and 2) second, to better handle data with complex distributions by the use of locally adaptive decision constraints, which can assess the similarity between a pair of samples based on the distance changes before and after metric learning. The experimental results obtained with a number of challenging hyperspectral image datasets demonstrate that the proposed LADRml algorithm outperforms the state-of-the-art dimensionality reduction and metric learning methods.

Automatic pattern recognition of ECG signals using entropy-based adaptive dimensionality reduction and clustering

In order to automatically recognize the patterns of ECG signals for different subjects, a new entropy-based principal component analysis (EPCA) is developed in this paper for dimensionality reduction of ECG signals. With the EPCA, Pb, the best number of principal components for a specific subject, is automatically determined by the energy to entropy ratio of reconstructed ECG signals after dimensionality reduction. Then, a novel fuzzy-entropy based c-means clustering (FECM) is proposed for cluster partition of ECG feature data. The optimal number of clusters, i.e. kb for a data set with specific ECG feature is found through a fuzzy-entropy based clustering measure (ECM), in which the average symmetric fuzzy cross entropy of membership subset pairs is combined with the average fuzzy entropy of clusters. Afterwards, the ECG signals in the MIT-BIH Arrhythmia Database are used for performance evaluation of EPCA and FECM. Five indices of signal reconstruction are employed to evaluate the results of dimensionality reduction, which shows that the EPCA performs better than the schemes based on cumulative percentage and scree graph when searching for Pb. Moreover, by comparing ECM with the other eight fuzzy clustering indices, i.e. PC, PE, MPC, XB, FS, Kwon, FHV and PBMF, the ECM has demonstrates the superiority in searching for kb. By using ECM, the adaptability of FECM to various ECG data sets has been strengthened. Its clustering accuracy is superior to those of three commonly-used algorithms, i.e. spectral clustering (NJW), hierarchical agglomerative (HCA) and K-means clustering.

A note on the multiplicative gamma process

Adaptive dimensionality reduction in high-dimensional problems is a key topic in statistics. The multiplicative gamma process takes a relevant step in this direction, but improved studies on its properties are required to ease implementation. This note addresses such aim.

Adaptive Dimensionality Reduction with Semi-Supervision (AdDReSS): Classifying Multi-Attribute Biomedical Data

Medical diagnostics is often a multi-attribute problem, necessitating sophisticated tools for analyzing high-dimensional biomedical data. Mining this data often results in two crucial bottlenecks: 1) high dimensionality of features used to represent rich biological data and 2) small amounts of labelled training data due to the expense of consulting highly specific medical expertise necessary to assess each study. Currently, no approach that we are aware of has attempted to use active learning in the context of dimensionality reduction approaches for improving the construction of low dimensional representations. We present our novel methodology, AdDReSS (Adaptive Dimensionality Reduction with Semi-Supervision), to demonstrate that fewer labeled instances identified via AL in embedding space are needed for creating a more discriminative embedding representation compared to randomly selected instances. We tested our methodology on a wide variety of domains ranging from prostate gene expression, ovarian proteomic spectra, brain magnetic resonance imaging, and breast histopathology. Across these various high dimensional biomedical datasets with 100+ observations each and all parameters considered, the median classification accuracy across all experiments showed AdDReSS (88.7%) to outperform SSAGE, a SSDR method using random sampling (85.5%), and Graph Embedding (81.5%). Furthermore, we found that embeddings generated via AdDReSS achieved a mean 35.95% improvement in Raghavan efficiency, a measure of learning rate, over SSAGE. Our results demonstrate the value of AdDReSS to provide low dimensional representations of high dimensional biomedical data while achieving higher classification rates with fewer labelled examples as compared to without active learning.

Adaptive semi-supervised dimensionality reduction based on pairwise constraints weighting and graph optimizing

With the rapid growth of high dimensional data, dimensionality reduction is playing a more and more important role in practical data processing and analysing tasks. This paper studies semi-supervised dimensionality reduction using pairwise constraints. In this setting, domain knowledge is given in the form of pairwise constraints, which specifies whether a pair of instances belong to the same class (must-link constraint) or different classes (cannot-link constraint). In this paper, a novel semi-supervised dimensionality reduction method called adaptive semi-supervised dimensionality reduction (ASSDR) is proposed, which can get the optimized low dimensional representation of the original data by adaptively adjusting the weights of the pairwise constraints and simultaneously optimizing the graph construction. Experiments on UCI classification and image recognition show that ASSDR is superior to many existing dimensionality reduction methods.

Locally adaptive dimensionality reduction for indexing large time series databases

Similarity search in large time series databases has attracted much research interest recently. It is a difficult problem because of the typically high dimensionality of the data. The most promising solutions involve performing dimensionality reduction on the data, then indexing the reduced data with a multidimensional index structure. Many dimensionality reduction techniques have been proposed, including Singular Value Decomposition (SVD), the Discrete Fourier transform (DFT), and the Discrete Wavelet Transform (DWT). In this article, we introduce a new dimensionality reduction technique, which we call Adaptive Piecewise Constant Approximation (APCA). While previous techniques (e.g., SVD, DFT and DWT) choose a common representation for all the items in the database that minimizes the global reconstruction error, APCA approximates each time series by a set of constant value segments of varying lengths such that their individual reconstruction errors are minimal. We show how APCA can be indexed using a multidimensional index structure. We propose two distance measures in the indexed space that exploit the high fidelity of APCA for fast searching: a lower bounding Euclidean distance approximation, and a non-lower-bounding, but very tight, Euclidean distance approximation, and show how they can support fast exact searching and even faster approximate searching on the same index structure. We theoretically and empirically compare APCA to all the other techniques and demonstrate its superiority.

Locally adaptive dimensionality reduction for indexing large time series databases

Similarity search in large time series databases has attracted much research interest recently. It is a difficult problem because of the typically high dimensionality of the data.. The most promising solutions involve performing dimensionality reduction on the data, then indexing the reduced data with a multidimensional index structure. Many dimensionality reduction techniques have been proposed, including Singular Value Decomposition (SVD), the Discrete Fourier transform (DFT), and the Discrete Wavelet Transform (DWT). In this work we introduce a new dimensionality reduction technique which we call Adaptive Piecewise Constant Approximation (APCA). While previous techniques (e.g., SVD, DFT and DWT) choose a common representation for all the items in the database that minimizes the global reconstruction error, APCA approximates each time series by a set of constant value segments of varying lengths such that their individual reconstruction errors are minimal. We show how APCA can be indexed using a multidimensional index structure. We propose two distance measures in the indexed space that exploit the high fidelity of APCA for fast searching: a lower bounding Euclidean distance approximation, and a non-lower bounding, but very tight Euclidean distance approximation and show how they can support fast exact searching, and even faster approximate searching on the same index structure. We theoretically and empirically compare APCA to all the other techniques and demonstrate its superiority.