Effective Dimensionality Reduction Technique Research Articles

A feature selection method based on the Golden Jackal-Grey Wolf Hybrid Optimization Algorithm.

This paper proposes a feature selection method based on a hybrid optimization algorithm that combines the Golden Jackal Optimization (GJO) and Grey Wolf Optimizer (GWO). The primary objective of this method is to create an effective data dimensionality reduction technique for eliminating redundant, irrelevant, and noisy features within high-dimensional datasets. Drawing inspiration from the Chinese idiom "Chai Lang Hu Bao," hybrid algorithm mechanisms, and cooperative behaviors observed in natural animal populations, we amalgamate the GWO algorithm, the Lagrange interpolation method, and the GJO algorithm to propose the multi-strategy fusion GJO-GWO algorithm. In Case 1, the GJO-GWO algorithm addressed eight complex benchmark functions. In Case 2, GJO-GWO was utilized to tackle ten feature selection problems. Experimental results consistently demonstrate that under identical experimental conditions, whether solving complex benchmark functions or addressing feature selection problems, GJO-GWO exhibits smaller means, lower standard deviations, higher classification accuracy, and reduced execution times. These findings affirm the superior optimization performance, classification accuracy, and stability of the GJO-GWO algorithm.

MFILS: Tri-Selection via Convex and Nonconvex Regularizations.

In many real-world applications, data are represented by multiple instances and simultaneously associated with multiple labels. These data are always redundant and generally contaminated by different noise levels. As a result, several machine learning models fail to achieve good classification and find an optimal mapping. Feature selection, instance selection, and label selection are three effective dimensionality reduction techniques. Nevertheless, the literature was limited to feature and/or instance selection but has, to some extent, neglected label selection, which also plays an essential role in the preprocessing step, as label noises can adversely affect the performance of the underlying learning algorithms. In this article, we propose a novel framework termed multilabel Feature Instance Label Selection (mFILS) that simultaneously performs feature, instance, and label selections in both convex and nonconvex scenarios. To the best of our knowledge, this article offers, for the first time ever, a study using the triple and simultaneous selection of features, instances, and labels based on convex and nonconvex penalties in a multilabel scenario. Experimental results are built on some known benchmark datasets to validate the effectiveness of the proposed mFILS.

Learning Accurate Label-Specific Features From Partially Multilabeled Data.

Feature selection is an effective dimensionality reduction technique, which can speed up an algorithm and improve model performance such as predictive accuracy and result comprehensibility. The study of selecting label-specific features for each class label has attracted considerable attention since each class label might be determined by some inherent characteristics, where precise label information is required to guide label-specific feature selection. However, obtaining noise-free labels is quite difficult and impractical. In reality, each instance is often annotated by a candidate label set that comprises multiple ground-truth labels and other false-positive labels, termed partial multilabel (PML) learning scenario. Here, false-positive labels concealed in a candidate label set might induce the selection of false label-specific features while masking the intrinsic label correlations, which misleads the selection of relevant features and compromises the selection performance. To address this issue, a novel two-stage partial multilabel feature selection (PMLFS) approach is proposed, which elicits credible labels to guide accurate label-specific feature selection. First, the label confidence matrix is learned to help elicit ground-truth labels from the candidate label set via the label structure reconstruction strategy, each element of which indicates how likely a class label is ground truth. After that, based on distilled credible labels, a joint selection model, including label-specific feature learner and common feature learner, is designed to learn accurate label-specific features to each class label and common features for all class labels. Besides, label correlations are fused into the features selection process to facilitate the generation of an optimal feature subset. Extensive experimental results clearly validate the superiority of the proposed approach.

Dual-Graph-Regularization Constrained Nonnegative Matrix Factorization with Label Discrimination for Data Clustering

NONNEGATIVE matrix factorization (NMF) is an effective technique for dimensionality reduction of high-dimensional data for tasks such as machine learning and data visualization. However, for practical clustering tasks, traditional NMF ignores the manifold information of both the data space and feature space, as well as the discriminative information of the data. In this paper, we propose a semisupervised NMF called dual-graph-regularization-constrained nonnegative matrix factorization with label discrimination (DCNMFLD). DCNMFLD combines dual graph regularization and prior label information as additional constraints, making full use of the intrinsic geometric and discriminative structures of the data, and can efficiently enhance the discriminative and exclusionary nature of clustering and improve the clustering performance. The evaluation of the clustering experimental results on four benchmark datasets demonstrates the effectiveness of our new algorithm.

SemiFREE: Semisupervised Feature Selection With Fuzzy Relevance and Redundancy

Feature selection, as an effective dimensionality reduction technique, is favored in preprocessing data. However, most existing algorithms are solely liable for labeled or unlabeled data, whereas a limited portion of real-world data is annotated with labels. In this paper, we therefore propose a novel scheme named SemiFREE, i.e., Semi-supervised feature selection with Fuzzy RElevance and rEdundancy. Firstly, both labeled and unlabeled samples are assigned with fuzzy decisions that allow class membership to naturally express the fuzziness or uncertainty in data labeling. Secondly, sample similarities in feature space and fuzzy decision are captured to induce fuzzy information measures for redefining the feature relevance and redundancy. Finally, adhering to the principle of relevance-maximization and redundancy-minimization, SemiFREE leverages the forward sequential searching strategy to identify qualified features progressively. Extensive experiments demonstrate the superiority of SemiFREE in the presence of partially labeled data against some other well-established feature selection algorithms.

Self-adaptive weighted interaction feature selection based on robust fuzzy dominance rough sets for monotonic classification

Feature selection is an effective dimensionality reduction technique for classification tasks. Monotonic classification task (MCT) is a special classification task in which features and decision obey a monotonic constraint, which is widely used in various application fields. Most of the existing feature selection methods for MCTs are based on some dominance rough set models. However, these methods only consider the correlation between features and decisions, and ignore the interaction between features. Moreover, these dominance rough set models are not robust to noisy information. Thus, to make up for these two deficiencies, we propose a novel feature selection approach based on an improved robust fuzzy dominance rough set for MCTs. First, a β−precision fuzzy dominance neighborhood rough set (β−PFDNRS) with robustness is proposed. Second, a feature evaluation index is presented, which not only considers the correlation between features and decision, but also considers the interaction between features. Finally, a β−PFDNRS based self-adaptive weighted interaction feature selection (β−SWIFS) algorithm is designed to select a feature sequence. Extensive experiments are conducted on fifteen public datasets, and the results show that the β−PFDNRS model has good robustness and the proposed feature selection algorithm has an excellent classification performance.

Vertical federated learning-based feature selection with non-overlapping sample utilization

Vertical federated learning (VFL) is a privacy preserving collaborative machine learning technique designed for distributed learning scenarios in which data from different parties have overlap in the sample space. In this paper, a VFL method for feature selection, which is an effective dimensionality reduction technique that selects a subset of informative features from high-dimensional data by eliminating irrelevant and redundant features, is proposed. Because of the potential insufficiency of useful information for learning informative features and the difficulty in sharing raw data among parties due to the increasing awareness of data privacy protection, it is desirable to exploit information from multiple parties without raw data sharing. In this paper, we propose a VFL-based feature selection method that leverages deep learning models as well as complementary information from features in the same samples at multiple parties without data disclosure. In order to further improve feature selection performance, information of samples that do not have features appearing in all parties are also utilized. Promising results in extensive experiments show the effectiveness of the proposed approach in terms of collaborative feature selection without data sharing.

A novel update rule of HALS algorithm for nonnegative matrix factorization and Zangwill’s global convergence

Nonnegative Matrix Factorization (NMF) has attracted a great deal of attention as an effective technique for dimensionality reduction of large-scale nonnegative data. Given a nonnegative matrix, NMF aims to obtain two low-rank nonnegative factor matrices by solving a constrained optimization problem. The Hierarchical Alternating Least Squares (HALS) algorithm is a well-known and widely-used iterative method for solving such optimization problems. However, the original update rule used in the HALS algorithm is not well defined. In this paper, we propose a novel well-defined update rule of the HALS algorithm, and prove its global convergence in the sense of Zangwill. Unlike conventional globally-convergent update rules, the proposed one allows variables to take the value of zero and hence can obtain sparse factor matrices. We also present two stopping conditions that guarantee the finite termination of the HALS algorithm. The practical usefulness of the proposed update rule is shown through experiments using real-world datasets.

Different Routes or Methods of Application for Dimensionality Reduction in Multicenter Studies Databases

Technological progress and digital transformation, which began with Big Data and Artificial Intelligence (AI), are currently transforming ways of working in all fields, to support decision-making, particularly in multicenter research. This study analyzed a sample of 5178 hospital patients, suffering from exacerbation of chronic obstructive pulmonary disease (eCOPD). Because of differences in disease stages and progression, the clinical pathologies and characteristics of the patients were extremely diverse. Our objective was thus to reduce dimensionality by projecting the data onto a lower dimensional subspace. The results obtained show that principal component analysis (PCA) is the most effective linear technique for dimensionality reduction. Four patient profile groups are generated with similar affinity and characteristics. In conclusion, dimensionality reduction is found to be an effective technique that permits the visualization of early indications of clinical patterns with similar characteristics. This is valuable since the development of other pathologies (chronic diseases) over any given time period influences clinical parameters. If healthcare professionals can have access to such information beforehand, this can significantly improve the quality of patient care, since this type of study is based on a multitude of data-variables that can be used to evaluate and monitor the clinical status of the patient.

A Divisive Hierarchical Clustering Approach to Hyperspectral Band Selection

Band selection, considered as an effective dimensionality reduction technique for hyperspectral imagery (HSI), has become a hot topic for decades. Although various clustering-based methods have been applied to band selection, only a few studies explored the hierarchical structure among different spectral bands. And with regard to conventional hierarchical clustering, implemented in an agglomerative manner, both efficiency and accuracy of band selection still remain to rise. Moreover, the noise sensitivity is a defect inherent in the procedure of clustering. To address these issues, we propose a divisive hierarchical clustering approach (DHCA) to hyperspectral band selection. Inspired by divisive analysis, DHCA is designed to obtain any number of band subsets, which captures the intrinsic hierarchy of hyperspectral bands simultaneously. By introducing the local density into average dissimilarity, it can suppress the outliers clustering separately. Also, given the order of the spectrum, channel interval makes the similarity more rational among bands. Finally, we select a representative band in each cluster from the information viewpoint to ensure the band subset with a high quality. Extensive experiments on three real public HSI datasets fully validate the superiority of the proposed method against state-of-the-art competitors.

Riemannian proximal stochastic gradient descent for sparse 2DPCA

Sparse two-dimensional principal component analysis (2DPCA) is an effective dimensionality reduction and data processing technique, which has been successfully applied in face recognition. Sparse 2DPCA is based on 2D image matrices instead of 1D vectors so it can well exploit the structural information of image pixels. The sparse 2DPCA model is regarded as a non-convex and non-smooth composite optimization problem. The manifold proximal gradient algorithm (ManPG) that is efficient and robust can globally converge to an optimal point in solving this kind of problem on the Stiefel manifold, such as sparse PCA. ManPG uses the proximal gradient method for the non-smooth term of this problem, which yields a convex subproblem on the tangent space. By solving this subproblem, a descent direction on the tangent space is obtained. Then a new update point on the manifold is obtained via retraction. However, in solving large-scale problems, the computational complexity of ManPG is quite high since ManPG requires calculating the full gradient in each iteration. In order to reduce the computational complexity of each iteration, a more computationally efficient Riemannian proximal stochastic gradient descent algorithm (RPSGD) is proposed, which utilizes the stochastic gradient instead of the full gradient in ManPG. We prove that the iteration complexity of the proposed RPSGD is O(1/ϵ4) while it maintains the sublinear convergence for the large-scale sparse 2DPCA. The experimental results on face recognition and handwritten digits classification data sets show RPSGD has the advantages of small-time overhead, high test accuracy, good reconstructed image quality, and relatively sparse solution in solving the large-scale sparse 2DPCA.

Zero-Shot Feature Selection via Transferring Supervised Knowledge

Feature selection, an effective technique for dimensionality reduction, plays an important role in many machine learning systems. Supervised knowledge can significantly improve the performance. However, faced with the rapid growth of newly emerging concepts, existing supervised methods might easily suffer from the scarcity and validity of labeled data for training. In this paper, the authors study the problem of zero-shot feature selection (i.e., building a feature selection model that generalizes well to “unseen” concepts with limited training data of “seen” concepts). Specifically, they adopt class-semantic descriptions (i.e., attributes) as supervision for feature selection, so as to utilize the supervised knowledge transferred from the seen concepts. For more reliable discriminative features, they further propose the center-characteristic loss which encourages the selected features to capture the central characteristics of seen concepts. Extensive experiments conducted on various real-world datasets demonstrate the effectiveness of the method.

An efficient face recognition approach combining likelihood-based sufficient dimension reduction and LDA

In this paper, an efficient approach is proposed for face recognition (FR) under pose and illumination variations. It is based on combining likelihood-based sufficient dimension reduction (LSDR) and linear discriminant analysis (LDA) using different facial features. LDA is a well–established technique for dimensionality reduction, while LSDR is a relatively new supervised subspace learning method based on the key concept of sufficiency, and consists of estimating a central subspace for low-dimensional representation of facial images. We show, empirically, that this combination can obviously increase the separability between face classes. The facial features that were considered here are either engineered (hand-crafted) features (e.g., discrete shearlet transform coefficients) or learned features which are obtained by retraining a deep learning-based system called FaceNet. To assess the performance of the proposed approach, different classification methods were used during the evaluation phase, namely collaborative representation based classifier (CRC), KNN, linear SVM and three regression-based classifiers (LSRC, LRC and LDRC). The extensive experiments performed on four publicly available face databases – the extended Yale B, FERET, LFW and CMU Multi-PIE– have demonstrated that, on the one hand, LSDR is an effective and efficiency dimensionality reduction technique for face recognition. Particularly, it can be used to significantly increase the performance of a deep learning-based system such as FaceNet, mainly, when training samples are insufficient. On the other hand, its combination with LDA can outperform best individual face recognition algorithm based on LSDR or LDA. Furthermore, the proposed approach compares favorably to current state-of-the-art methods on face recognition, in terms of classification accuracy, under illumination and pose variations.

Exploiting Multilabel Information for Noise-Resilient Feature Selection

In a conventional supervised learning paradigm, each data instance is associated with one single class label. Multilabel learning differs in the way that data instances may belong to multiple concepts simultaneously, which naturally appear in a variety of high impact domains, ranging from bioinformatics and information retrieval to multimedia analysis. It targets leveraging the multiple label information of data instances to build a predictive learning model that can classify unlabeled instances into one or multiple predefined target classes. In multilabel learning, even though each instance is associated with a rich set of class labels, the label information could be noisy and incomplete as the labeling process is both time consuming and labor expensive, leading to potential missing annotations or even erroneous annotations. The existence of noisy and missing labels could negatively affect the performance of underlying learning algorithms. More often than not, multilabeled data often has noisy, irrelevant, and redundant features of high dimensionality. The existence of these uninformative features may also deteriorate the predictive power of the learning model due to the curse of dimensionality. Feature selection, as an effective dimensionality reduction technique, has shown to be powerful in preparing high-dimensional data for numerous data mining and machine-learning tasks. However, a vast majority of existing multilabel feature selection algorithms either boil down to solving multiple single-labeled feature selection problems or directly make use of the imperfect labels to guide the selection of representative features. As a result, they may not be able to obtain discriminative features shared across multiple labels. In this article, to bridge the gap between a rich source of multilabel information and its blemish in practical usage, we propose a novel noise-resilient multilabel informed feature selection framework (MIFS) by exploiting the correlations among different labels. In particular, to reduce the negative effects of imperfect label information in obtaining label correlations, we decompose the multilabel information of data instances into a low-dimensional space and then employ the reduced label representation to guide the feature selection phase via a joint sparse regression framework. Empirical studies on both synthetic and real-world datasets demonstrate the effectiveness and efficiency of the proposed MIFS framework.

Generalized leaky integrate-and-fire models classify multiple neuron types

There is a high diversity of neuronal types in the mammalian neocortex. To facilitate construction of system models with multiple cell types, we generate a database of point models associated with the Allen Cell Types Database. We construct a set of generalized leaky integrate-and-fire (GLIF) models of increasing complexity to reproduce the spiking behaviors of 645 recorded neurons from 16 transgenic lines. The more complex models have an increased capacity to predict spiking behavior of hold-out stimuli. We use unsupervised methods to classify cell types, and find that high level GLIF model parameters are able to differentiate transgenic lines comparable to electrophysiological features. The more complex model parameters also have an increased ability to differentiate between transgenic lines. Thus, creating simple models is an effective dimensionality reduction technique that enables the differentiation of cell types from electrophysiological responses without the need for a priori-defined features. This database will provide a set of simplified models of multiple cell types for the community to use in network models.

A Hybrid Approach for Feature Subset Selection using Ant Colony Optimization and Multi-Classifier Ensemble

An active area of research in data mining and machine learning is dimensionality reduction. Feature subset selection is an effective technique for dimensionality reduction and an essential step in successful data mining applications. It reduces the number of features, removes irrelevant, redundant, or noisy features, and enhances the predictive capability of the classifier. It provides fast and cost-effective predictors and leading to better model comprehensibility. In this paper, we proposed a hybrid approach for feature subset selection. It is a filter based method in which a classifier ensemble is coupled with Ant colony optimization algorithm to enhance the predictive accuracy of filters. Extensive experimentation has been carried out on eleven data sets over four different classifiers. All of the data sets are available publically. We have compared our proposed method with numerous filter and wrapper based methods. Experimental results indicate that our method has remarkable ability to generate subsets with reduced number of features. Along with it, our proposed method attained higher classification accuracy.

Sparsity and Geometry Preserving Graph Embedding for Dimensionality Reduction

Graph embedding is a very useful dimensionality reduction technique in pattern recognition. In this paper, we develop a novel discriminative dimensionality reduction technique entitled sparsity and geometry preserving graph embedding (SGPGE). SGPGE can not only capture the sparse reconstructive relationships among training samples but also discover the intrinsic geometry and latent discrimination from high-dimensional data. In SGPGE, the novel ways of constructing global and local adjacent graphs are developed. The built graphs with discriminant and geometrical information are more informative in graph embedding. Integrating the sparse reconstruction with the designed local and global adjacent graph constructions, SGPGE newly characterizes with-class sparsity and geometry preserving scatter, between-class sparsity and geometry preserving scatter and local sparsity and geometry preserving scatter to formulate the objective function of dimensionality reduction, and learns the discriminative transformation matrix using maximum margin criterion. Moreover, we also propose three variants of SGPGE. Experimental results on six public face datasets have demonstrated that the proposed methods are effective dimensionality reduction techniques with very good performance for classification in the embedding subspace.

Multimodal, high-dimensional, model-based, Bayesian inverse problems with applications in biomechanics

This paper is concerned with the numerical solution of model-based, Bayesian inverse problems. We are particularly interested in cases where the cost of each likelihood evaluation (forward-model call) is expensive and the number of unknown (latent) variables is high. This is the setting in many problems in computational physics where forward models with nonlinear PDEs are used and the parameters to be calibrated involve spatio-temporarily varying coefficients, which upon discretization give rise to a high-dimensional vector of unknowns.One of the consequences of the well-documented ill-posedness of inverse problems is the possibility of multiple solutions. While such information is contained in the posterior density in Bayesian formulations, the discovery of a single mode, let alone multiple, poses a formidable computational task. The goal of the present paper is two-fold. On one hand, we propose approximate, adaptive inference strategies using mixture densities to capture multi-modal posteriors. On the other, we extend our work in [1] with regard to effective dimensionality reduction techniques that reveal low-dimensional subspaces where the posterior variance is mostly concentrated. We validate the proposed model by employing Importance Sampling which confirms that the bias introduced is small and can be efficiently corrected if the analyst wishes to do so. We demonstrate the performance of the proposed strategy in nonlinear elastography where the identification of the mechanical properties of biological materials can inform non-invasive, medical diagnosis. The discovery of multiple modes (solutions) in such problems is critical in achieving the diagnostic objectives.

Robust kernel discriminant analysis and its application to feature extraction and recognition

Subspace analysis is an effective technique for dimensionality reduction, which aims at finding a low-dimensional space of high-dimensional data. In this paper, a novel subspace method called robust kernel discriminant analysis is proposed for dimensionality reduction. An optimization function is firstly defined in terms of the distance between similar elements and the distance between dissimilar elements, which can preserve the structure of the data in the mapping space. Then the optimization function is transformed into an eigenvalue problem and the projection vectors are obtained by solving the eigenvalue problem. Finally, experimental results on face images and handwritten numerical characters demonstrate the effectiveness and feasibility of the proposed method.