Hierarchical Feature Selection Method Research Articles

HFS-CSR: A hierarchical feature selection method based on correlation and structural redundancy

Feature selection (FS) is a crucial step in data engineering, as it not only saves time and resources for model training but also improves the prediction performance. In many realistic data sets, the feature space is formed by binary features which are related via generalization-specialization relationships (a.k.a., hierarchical feature space). These interdependencies among features, which can be described via a directed acyclic graph, present new challenges for classical FS methods. In this paper, a Hierarchical Feature Selection method based on Correlation and Structural Redundancy (HFS-CSR) is proposed to address it. In HFS-CSR, the correlation between features and the class label is measured based on Fisher Score whereas the structural redundancy among features are assessed quantitatively via hierarchical redundancy and depth penalties. The importance of features are determined by combining them adaptively for feature filtering. Experiments are conducted based on two kinds of typical data sets with hierarchical feature space: aging gene data sets and NLP data sets. The results demonstrate that HFS-CSR can achieve better performance compared with seven existing FS methods.

A Novel Hierarchical Clustering Sequential Forward Feature Selection Method for Paddy Rice Agriculture Mapping Based on Time-Series Images

Timely and accurate mapping of rice distribution is crucial to estimate yield, optimize agriculture spatial patterns, and ensure global food security. Feature selection (FS) methods have significantly improved computational efficiency by reducing redundancy in spectral and temporal feature sets, playing a vital role in identifying and mapping paddy rice. However, the optimal feature sets selected by existing methods suffer from issues such as information redundancy or local optimality, limiting their accuracy in rice identification. Moreover, the effects of these FS methods on rice recognition in various machine learning classifiers and regions with different climatic conditions and planting structures is still unclear. To overcome these limitations, we conducted a comprehensive evaluation of the potential applications of major FS methods, including the wrapper method, embedded method, and filter method for rice mapping. A novel hierarchical lustering sequential forward selection (HCSFS) method for precisely extracting the optimal feature set for rice identification is proposed. The accuracy of the HCSFS and other FS methods for rice identification was tested with nine common machine learning classifiers. The results indicated that, among the three FS methods, the wrapper method achieved the best rice mapping performance, followed by the embedded method, and lastly, the filter method. The new HCSFS significantly reduced redundant features compared with eleven typical FS methods, demonstrating higher precision and stability, with user accuracy and producer accuracy exceeding 0.9548 and 0.9487, respectively. Additionally, the spatial distribution of rice maps generated using the optimal feature set selected by HCSFS closely aligned with actual planting patterns, markedly outperforming existing rice products. This research confirms the effectiveness and transferability of the HCSFS method for rice mapping across different climates and cultivation structures, suggesting its enormous potential for classifying other crops using time-series remote sensing images.

A hierarchical attention-based feature selection and fusion method for credit risk assessment

A stable financial environment is requisite for the continuous growth of the E-business market, emphasizing the importance of credit risk assessment. Generally, credit risk assessment involves evaluating the creditworthiness of the economic entity and predicting the probability of defaulting on their financial obligations. Existing works mainly focus on exploring more features from various sources to help default prediction, but tend to overlook the feature cost and feature fusion problem. In this paper, we proposed a novel credit risk assessment model based on the hierarchical attention method, characterizing the ability to manage feature acquisition cost and adaptively integrate multi-view features. Particularly, to fulfill the utility of multi-source features, three segmenting standards are proposed to categorize multi-source features into different views according to their characteristics and statistical attributes. Then, a feature-level attention mechanism is proposed to estimate feature importance and guide feature selection, while a view-level attention mechanism is proposed to aggregate view representations and produce default predictions. The proposed method is capable of enhancing crucial features in a hierarchical way and undertaking feature selection to meet feature acquisition cost, thereby boosting the model performance with feature cost awareness. Experimental results show that the proposed method can approximate the best performance with 5% of selected features, and can also achieve a 1.1% AUC improvement compared with the best baseline method over the full feature set, demonstrating our effectiveness of enhancing default prediction performance while reducing feature costs.

HHFS: A Hybrid Hierarchical Feature Selection Method for Ageing Gene Classification

As one of the most complicated processes in biological development, ageing remains poorly understood. These days more and more ageing-related gene datasets become available on the web, where each instance is characterized by a set of hierarchically-organized binary features. Traditional data mining methods show limitations in exploiting this hierarchical feature space. This paper proposes a hybrid hierarchical feature selection (HHFS) method for classifying genes into pro-longevity or anti-longevity ones. HHFS conducts lazy and eager feature selections sequentially, taking into account both uniqueness of a test instance and the whole characteristics of datasets. It adopts two complementary relevancy metrics (i.e., Gini purity and mutual information) to remove hierarchical redundancy. The experiments are conducted based on the ageing-related gene data of four model organisms. The results show that HHFS achieves significantly better prediction performance than several state-of-the-art methods.

Semantic-gap-oriented feature selection in hierarchical classification learning

Semantic gap is common issues in classification tasks. Different from traditional classification tasks, the semantic gap problem can introduce serious error propagation for hierarchical classification. In this paper, we propose a hierarchical feature selection method with label enhancement (HFSLE) to solve this challenge, which estimates the label distribution and selects subsets of features for each subtask. Firstly, HFSLE combines the label enhancement and hierarchical feature selection process so that the label enhancement benefits from the feedback of the loss function. Then, the label enhancement is constrained by the consistency of the parent-child distribution to obtain a plausible label distribution. Finally, feature selection is optimized based on intra-class consistency hierarchical regularization terms, which unifies feature subsets with hierarchical semantic descriptions to narrow the semantic gap. HFSLE gives a new perspective on the hierarchical feature selection based on label distribution rather than the original logical label. To demonstrate the superiority of the method, we compared it with six well-established feature selection methods on eight hierarchical data sets, and analyzed the convergence and robustness of the method. Extensive experiments demonstrate that the proposed method can perform favorably against advanced hierarchical feature selection approaches.

Feature selection via maximizing inter-class independence and minimizing intra-class redundancy for hierarchical classification

Hierarchical feature selection has proven to be significant in reducing classification difficulty. Many existing hierarchical feature selection methods use the hierarchy in the class space as structural information to improve performance. However, these methods focus on the inter-class structural relations and ignore the intra-class feature correlations, which is challenging as the numbers of classes and features in current datasets are both growing rapidly. In this paper, we propose a hierarchical feature selection method that maximizes inter-class independence and minimizes intra-class redundancy using structure and feature relations. First, we investigate the class hierarchy dependency in the tree structure as the structural relation regularization, which maximizes the independence between unrelated classes in the class structure. Second, we transform the feature correlations into a mathematical representation as feature relation regularization, which minimizes the redundancy in each class on the premise of sparsity. Finally, we unify the two regularization terms into a hierarchical feature selection method to trade off the structural and feature relations. Our method exhibits excellent efficiency and effectiveness compared with other feature selection methods.

Performance Analysis of Coal Gangue Recognition Based on Hierarchical Filtering and Coupled Wrapper Feature Selection Method

Performance Analysis of Coal Gangue Recognition Based on Hierarchical Filtering and Coupled Wrapper Feature Selection Method

Hierarchical unsupervised multi-view feature selection

Multi-view feature selection is an important research direction in the field of multi-view learning. To remove the negative effect of irrelevant and redundant information and select the important features from the multi-view data, in this paper, we propose a novel hierarchical unsupervised multi-view feature selection method, which integrates matrix decomposition and hierarchical regularization as a joint model. Specifically, to exploit the consistency information of multiple views for projection matrix, we project the multiple views into a latent basis space based on a matrix decomposition model. The hierarchical regularization containing the [Formula: see text]-norm, dependence and Frobenius-norm is imposed on projection matrix to, respectively, exploit the row-level, dependency-level and view-level feature selection. We also develop an efficient optimization algorithm to optimize our method. Extensive experimental results on six popular multi-view datasets show the effectiveness and superiority of our method by comparing with the state-of-the-art methods.

A novel hierarchical feature selection method based on large margin nearest neighbor learning

The complexity of data has been increasing not only in the dimension of the data, but also in the interrelated class relationships. Pre-defined hierarchical structure of classes helps to model such relationships in a way, which divides the whole classification problem into a set of subtasks. Feature selection, known as a data preprocessing technique, aims to selecting useful feature to improve the performance of learning algorithms. However, the issue of close class relationships, especially under the same parent node in the hierarchical structure, is adverse to the feature selection. In this paper, we introduce a novel distance metric based feature selection method with large margin nearest neighbor strategy for the hierarchical classification problems. Different from other algorithms, our proposed method is to project the data into a low-dimensional feature space for learning correct metrics, and the weights of features will be indicated in the transformation matrix. A sparsity regularization and iterative optimization ensure that superior feature subsets can be selected. Experimental evaluation on several datasets with hierarchical class structure demonstrates that the proposed approach is comparable or better than some other state-of-the-art hierarchical feature selection methods.

Transmission Line Fault-Cause Identification Based on Hierarchical Multiview Feature Selection

Fault-cause identification plays a significant role in transmission line maintenance and fault disposal. With the increasing types of monitoring data, i.e., micrometeorology and geographic information, multiview learning can be used to realize the information fusion for better fault-cause identification. To reduce the redundant information of different types of monitoring data, in this paper, a hierarchical multiview feature selection (HMVFS) method is proposed to address the challenge of combining waveform and contextual fault features. To enhance the discriminant ability of the model, an ε-dragging technique is introduced to enlarge the boundary between different classes. To effectively select the useful feature subset, two regularization terms, namely l2,1-norm and Frobenius norm penalty, are adopted to conduct the hierarchical feature selection for multiview data. Subsequently, an iterative optimization algorithm is developed to solve our proposed method, and its convergence is theoretically proven. Waveform and contextual features are extracted from yield data and used to evaluate the proposed HMVFS. The experimental results demonstrate the effectiveness of the combined used of fault features and reveal the superior performance and application potential of HMVFS.

HFS-SLPEE: A Novel Hierarchical Feature Selection and Second Learning Probability Error Ensemble Model for Precision Cancer Diagnosis.

The emergence of high-throughput RNA-seq data has offered unprecedented opportunities for cancer diagnosis. However, capturing biological data with highly nonlinear and complex associations by most existing approaches for cancer diagnosis has been challenging. In this study, we propose a novel hierarchical feature selection and second learning probability error ensemble model (named HFS-SLPEE) for precision cancer diagnosis. Specifically, we first integrated protein-coding gene expression profiles, non-coding RNA expression profiles, and DNA methylation data to provide rich information; afterward, we designed a novel hierarchical feature selection method, which takes the CpG-gene biological associations into account and can select a compact set of superior features; next, we used four individual classifiers with significant differences and apparent complementary to build the heterogeneous classifiers; lastly, we developed a second learning probability error ensemble model called SLPEE to thoroughly learn the new data consisting of classifiers-predicted class probability values and the actual label, further realizing the self-correction of the diagnosis errors. Benchmarking comparisons on TCGA showed that HFS-SLPEE performs better than the state-of-the-art approaches. Moreover, we analyzed in-depth 10 groups of selected features and found several novel HFS-SLPEE-predicted epigenomics and epigenetics biomarkers for breast invasive carcinoma (BRCA) (e.g., TSLP and ADAMTS9-AS2), lung adenocarcinoma (LUAD) (e.g., HBA1 and CTB-43E15.1), and kidney renal clear cell carcinoma (KIRC) (e.g., IRX2 and BMPR1B-AS1).

Hierarchical feature selection with multi-granularity clustering structure

Hierarchical feature selection addresses the issues caused by the presence of high-dimensional features in multi-category classification systems with hierarchical structures. Granular calculations are made to analyze the hierarchical relationships among categories when selecting the optimal feature subset. However, semantic hierarchy-based feature selection methods are prone to the semantic gap problem, which affects classification accuracy. In this paper, we propose a hierarchical feature selection method with a multi-granularity clustering structure that can effectively alleviate the semantic gap problem. Firstly, a hierarchical structure is constructed via bottom-up multi-granularity clustering based on feature similarities rather than semantic categories. This clustering hierarchy is conducive to solving semantic gap problems in the existing hierarchy. Secondly, the optimal feature subset is selected using the ℓ1,2-norms in each hierarchy’s granularity layer. This joint minimization approach can retain both the granularity layers’ shared features and granularity-specific features. Finally, we execute hierarchical classification according to the granular structure in a coarse to fine sequence. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art hierarchical feature selection approaches.

Harvestman: a framework for hierarchical feature learning and selection from whole genome sequencing data

BackgroundSupervised learning from high-throughput sequencing data presents many challenges. For one, the curse of dimensionality often leads to overfitting as well as issues with scalability. This can bring about inaccurate models or those that require extensive compute time and resources. Additionally, variant calls may not be the optimal encoding for a given learning task, which also contributes to poor predictive capabilities. To address these issues, we present Harvestman, a method that takes advantage of hierarchical relationships among the possible biological interpretations and representations of genomic variants to perform automatic feature learning, feature selection, and model building.ResultsWe demonstrate that Harvestman scales to thousands of genomes comprising more than 84 million variants by processing phase 3 data from the 1000 Genomes Project, one of the largest publicly available collection of whole genome sequences. Using breast cancer data from The Cancer Genome Atlas, we show that Harvestman selects a rich combination of representations that are adapted to the learning task, and performs better than a binary representation of SNPs alone. We compare Harvestman to existing feature selection methods and demonstrate that our method is more parsimonious—it selects smaller and less redundant feature subsets while maintaining accuracy of the resulting classifier.ConclusionHarvestman is a hierarchical feature selection approach for supervised model building from variant call data. By building a knowledge graph over genomic variants and solving an integer linear program , Harvestman automatically and optimally finds the right encoding for genomic variants. Compared to other hierarchical feature selection methods, Harvestman is faster and selects features more parsimoniously.

Robust hierarchical feature selection with a capped [formula omitted]-norm

Feature selection methods face new challenges in large-scale classification tasks because massive categories are managed in a hierarchical structure. Hierarchical feature selection can take full advantage of the dependencies among hierarchically structured classes. However, most of the existing hierarchical feature selection methods are not robust for dealing with the inevitable data outliers, resulting in a serious inter-level error propagation problem in the following classification process. In this paper, we propose a robust hierarchical feature selection method with a capped ℓ2-norm (HFSCN), which can reduce the adverse effects of data outliers and learn relatively robust and discriminative feature subsets for the hierarchical classification process. Firstly, a large-scale global classification task is split into several small local sub-classification tasks according to the hierarchical class structure and the divide-and-conquer strategy, which makes it easy for feature selection modeling. Secondly, a capped ℓ2-norm based loss function is used in the feature selection process of each local sub-classification task to eliminate the data outliers, which can alleviate the negative effects outliers and improve the robustness of the learned feature weighted matrix. Finally, an inter-level relation constraint based on the similarity between the parent and child classes is added to the feature selection model, which can enhance the discriminative ability of the selected feature subset for each sub-classification task with the learned robust feature weighted matrix. Compared with seven traditional and state-of-art hierarchical feature selection methods, the superior performance of HFSCN is verified on 16 real and synthetic datasets.

Robust hierarchical feature selection driven by data and knowledge

Feature selection is facing great challenges brought by the enlarging label space and the inevitable noisy data. Flat feature selection methods fail to obtain a compact feature subset because of the numerous classes. In addition, these data-driven methods are sensitive to the data outliers. Fortunately, many practical tasks usually organize the classes by a hierarchical structure in a coarse-to-fine manner and can be solved by using the divide-and-conquer strategy. In this paper, we propose a hierarchical feature selection method driven by data and knowledge (HFSDK), which is robust to the data outliers and produces compact feature subsets by splitting the original large label space. Firstly, HFSDK decomposes a large-scale classification task into a group of small subclassification tasks with multiple granularities, which is driven by knowledge of the hierarchical class structure. Then, the corresponding datasets are constructed from the bottom to the top using the class labels of data, which is a data-driven process. Finally, robust and discriminative feature subsets are selected recursively for those subtasks by eliminating the data outliers and adding a semantic relation constraint. Experiments on six real-world datasets validate the superior performance of the proposed method.

Prioritizing positive feature values: a new hierarchical feature selection method

In this work we address the problem of feature selection for the classification task in hierarchical and sparse feature spaces, which characterize many real-world applications nowadays. A binary feature space is deemed hierarchical when its binary features are related via generalization-specialization relationships, and is considered sparse when in general the instances contain much fewer “positive” than “negative” feature values. In any given instance, a feature value is deemed positive (negative) when the property associated with the feature has been (has not been) observed for that instance. Although there are many methods for the traditional feature selection problem in the literature, the proper treatment to hierarchical feature structures is still a challenge. Hence, we introduce a novel hierarchical feature selection method that follows the lazy learning paradigm—selecting a feature subset tailored for each instance in the test set. Our strategy prioritizes the selection of features with positive values, since they tend to be more informative—the presence of a relatively rare property is usually a piece of more relevant information than the absence of that property. Experiments on different application domains have shown that the proposed method outperforms previous hierarchical feature selection methods and also traditional methods in terms of predictive accuracy, selecting smaller feature subsets in general.

An empirical evaluation of hierarchical feature selection methods for classification in bioinformatics datasets with gene ontology-based features

Hierarchical feature selection is a new research area in machine learning/data mining, which consists of performing feature selection by exploiting dependency relationships among hierarchically structured features. This paper evaluates four hierarchical feature selection methods, i.e., HIP, MR, SHSEL and GTD, used together with four types of lazy learning-based classifiers, i.e., Naive Bayes, Tree Augmented Naive Bayes, Bayesian Network Augmented Naive Bayes and k-Nearest Neighbors classifiers. These four hierarchical feature selection methods are compared with each other and with a well-known “flat” feature selection method, i.e., Correlation-based Feature Selection. The adopted bioinformatics datasets consist of aging-related genes used as instances and Gene Ontology terms used as hierarchical features. The experimental results reveal that the HIP (Select Hierarchical Information Preserving Features) method performs best overall, in terms of predictive accuracy and robustness when coping with data where the instances’ classes have a substantially imbalanced distribution. This paper also reports a list of the Gene Ontology terms that were most often selected by the HIP method.

Predicting the Pro-Longevity or Anti-Longevity Effect of Model Organism Genes with New Hierarchical Feature Selection Methods.

Ageing is a highly complex biological process that is still poorly understood. With the growing amount of ageing-related data available on the web, in particular concerning the genetics of ageing, it is timely to apply data mining methods to that data, in order to try to discover novel patterns that may assist ageing research. In this work, we introduce new hierarchical feature selection methods for the classification task of data mining and apply them to ageing-related data from four model organisms: Caenorhabditis elegans (worm), Saccharomyces cerevisiae (yeast), Drosophila melanogaster (fly), and Mus musculus (mouse). The main novel aspect of the proposed feature selection methods is that they exploit hierarchical relationships in the set of features (Gene Ontology terms) in order to improve the predictive accuracy of the Naïve Bayes and 1-Nearest Neighbour (1-NN) classifiers, which are used to classify model organisms' genes into pro-longevity or anti-longevity genes. The results show that our hierarchical feature selection methods, when used together with Naïve Bayes and 1-NN classifiers, obtain higher predictive accuracy than the standard (without feature selection) Naïve Bayes and 1-NN classifiers, respectively. We also discuss the biological relevance of a number of Gene Ontology terms very frequently selected by our algorithms in our datasets.

A Hierarchical Feature Selection Method Based on Classification Tree for HGU Fault Diagnosis

Fault diagnosis is very important to ensure the safe operation of hydraulic generator units (HGU). Because of the complexity of HGU, the vast amounts of measured data and the redundant information, the accuracy and instantaneity of fault diagnosis are severely limited. At present, feature selection technique is an effective method to break through this bottleneck. According to the specific characteristics of HGU faults, this paper puts forward a hierarchical feature selection method based on classification tree (HFSMCT). HFSMCT selects the most effective feature for each branch node through filtering evaluation criteria and heuristic search strategy, and all the selected features constitute the final feature set. Moreover, HFSMCT is easy to design and implement, and it is very prominent in computational efficiency and accuracy. The simulation results also prove that HFSMCT is very suitable for HGU fault diagnosis.

Hierarchical feature selection method for detection of obfuscated malicious code

Obfuscated malicious codes can easily escape from being detected by the conventional static method.On the other hand,despite its high detection accuracy,the dynamic method usually expends a large amount of system resources.A hierarchical feature selection method was proposed to improve the detection accuracy with relative low system overhead,where the features were generated and subsequently selected on the oriented layer,the individual layer,the family layer and the global layer,respectively.By the layer-by-layer refinements,an appropriate trade-off between the feature redundancy and information omission was archived using the hierarchical feature selection method.The experimental results on the real-world datasets demonstrate that the proposed method yields high accuracy for detecting obfuscated malicious code,while has several advantages such as smaller size of required training samples and better generalization ability compared with the conventional feature selection methods.