Step In Machine Learning Research Articles

Joint adaptive manifold and embedding learning for unsupervised feature selection

As data always lie on a lower-dimensional space, feature selection has become an important step in computer vision, machine learning and data mining. Due to the lack of class information, the performance of unsupervised feature selection depends on how to characterize and preserve the manifold structure among data. In this paper, we propose a novel unsupervised feature selection framework, named as joint adaptive manifold and embedding learning for unsupervised feature selection (JAMEL). It iteratively and adaptively learns lower-dimensional embeddings for data to preserve the manifold structure among data, regresses data to embeddings to measure the importance of features, and learns the manifold structure among data according to the data density in the intrinsic space, where the redundant and noisy features are eliminated. In addition, we present an efficient algorithm to solve the proposed problem, together with the convergence analysis. Finally, the evaluation results with the tasks of k-means, spectral clustering and nearest neighbor classification using the selected features on 12 datasets show the effectiveness and efficiency of our approach.

Cost-sensitive feature selection on multi-label data via neighborhood granularity and label enhancement

Multi-label feature selection, which is an efficient and effective pre-processing step in machine learning and data mining, can select a feature subset that contains more contributions for multi-label classification while improving the performance of the classifiers. In real-world applications, an instance may be associated with multiple related labels with different relative importances, and the process of obtaining different features usually requires different costs, containing money, and time, etc. However, most existing works with regard to multi-label feature selection do not take into consideration the above two critical issues simultaneously. Therefore, in this paper, we exploit the idea of neighborhood granularity to enhance the traditional logical labels into label distribution forms to excavate the deeper supervised information hidden in multi-label data, and further consider the effect of the test cost under three different distributions, simultaneously. Motivated by these issues, a novel test cost multi-label feature selection algorithm with label enhancement and neighborhood granularity is designed. Moreover, the proposed algorithm is tested upon ten publicly available benchmark multi-label datasets with six widely-used metrics from two different aspects. Finally, two groups of experimental results demonstrate that the proposed algorithm achieves the satisfactory and superior performance over other four state-of-the-art comparing algorithms, and it is effective for improving the learning performance and decreasing the total test costs of the selected feature subset.

A new approach of attribute reduction of rough sets based on soft metric

Attribute reduction is considered as an important processing step for pattern recognition, machine learning and data mining. In this paper, we combine soft set and rough set to use them in applications. We generalize rough set model and introduce a soft metric rough set model to deal with the problem of heterogeneous numerical feature subset selection. We construct a soft metric on the family of knowledge structures based on the soft distance between attributes. The proposed model will degrade to the classical one if we specify a zero soft real number. We also provide a systematic study of attribute reduction of rough sets based on soft metric. Based on the constructed metric, we define co-information systems and consistent co-decision systems, and we provide a new method of attribute reductions of each system. Furthermore, we present a judgement theorem and discernibility matrix associated with attribute of each type of system. As an application, we present a case study from Zoo data set to verify our theoretical results.

A two-layer feature selection method using Genetic Algorithm and Elastic Net

Feature selection, as a critical pre-processing step for machine learning, aims at determining representative predictors from a high-dimensional feature space dataset to improve the prediction accuracy. However, the increase in feature space dimensionality, comparing to the number of observations, poses a severe challenge to many existing feature selection methods considering computational efficiency and prediction performance. This paper presents a new two-layer feature selection approach that combines a wrapper and an embedded method in constructing an appropriate subset of predictors. In the first layer of the proposed method, Genetic Algorithm (GA) has been adopted as a wrapper to search for the optimal subset of predictors, which aims to reduce the number of predictors and the prediction error. As one of the meta-heuristic approaches, GA is selected due to its computational efficiency; however, GAs do not guarantee the optimality. To address this issue, a second layer is added to the proposed method to eliminate any remaining redundant/irrelevant predictors to improve the prediction accuracy. Elastic Net (EN) has been selected as the embedded method in the second layer because of its flexibility in adjusting the penalty terms in the regularization process and time efficiency. This two-layer approach has been applied on a Maize genetic dataset from NAM population, which consists of multiple subsets of datasets with different ratios of the number of predictors to the number of observations. The numerical results confirm the superiority of the proposed model.

Causality-based Feature Selection

Feature selection is a crucial preprocessing step in data analytics and machine learning. Classical feature selection algorithms select features based on the correlations between predictive features and the class variable and do not attempt to capture causal relationships between them. It has been shown that the knowledge about the causal relationships between features and the class variable has potential benefits for building interpretable and robust prediction models, since causal relationships imply the underlying mechanism of a system. Consequently, causality-based feature selection has gradually attracted greater attentions and many algorithms have been proposed. In this article, we present a comprehensive review of recent advances in causality-based feature selection. To facilitate the development of new algorithms in the research area and make it easy for the comparisons between new methods and existing ones, we develop the first open-source package, called CausalFS, which consists of most of the representative causality-based feature selection algorithms (available at https://github.com/kuiy/CausalFS). Using CausalFS, we conduct extensive experiments to compare the representative algorithms with both synthetic and real-world datasets. Finally, we discuss some challenging problems to be tackled in future research.

Learning comprehensible and accurate hybrid trees

Finding the best classifiers according to different criteria is often performed by a multi-objective machine learning algorithm. This study considers two criteria that are usually treated as the most important when deciding which classifier to apply in practice: comprehensibility and accuracy. A model that offers a broad range of trade-offs between the two criteria is introduced because they conflict; i.e., increasing one decreases the other. The choice of the model is motivated by the fact that domain experts often formalize decisions based on knowledge that can be represented by comprehensible rules and some tacit knowledge. This approach is mimicked by a hybrid tree that consists of comprehensible parts that originate from a regular classification tree and incomprehensible parts that originate from an accurate black-box classifier. An empirical evaluation on 23 UCI datasets shows that the hybrid trees provide trade-offs between the accuracy and comprehensibility that are not possible using traditional machine learning models. A corresponding hybrid-tree comprehensibility metric is also proposed. Furthermore, the paper presents a novel algorithm for learning MAchine LeArning Classifiers with HybrId TrEes (MALACHITE), and it proves that the algorithm finds a complete set of nondominated hybrid trees with regard to their accuracy and comprehensibility. The algorithm is shown to be faster than the well-known multi-objective evolutionary optimization algorithm NSGA-II for trees with moderate size, which is a prerequisite for comprehensibility. On the other hand, the MALACHITE algorithm can generate considerably larger hybrid-trees than a naïve exhaustive search algorithm in a reasonable amount of time. In addition, an interactive iterative data mining process based on the algorithm is proposed that enables inspection of the Pareto set of hybrid trees. In each iteration, the domain expert analyzes the current set of nondominated hybrid trees, infers domain relations, and sets the parameters for the next machine learning step accordingly.

Interpretation of omics data analyses

Omics studies attempt to extract meaningful messages from large-scale and high-dimensional data sets by treating the data sets as a whole. The concept of treating data sets as a whole is important in every step of the data-handling procedures: the pre-processing step of data records, the step of statistical analyses and machine learning, translation of the outputs into human natural perceptions, and acceptance of the messages with uncertainty. In the pre-processing, the method by which to control the data quality and batch effects are discussed. For the main analyses, the approaches are divided into two types and their basic concepts are discussed. The first type is the evaluation of many items individually, followed by interpretation of individual items in the context of multiple testing and combination. The second type is the extraction of fewer important aspects from the whole data records. The outputs of the main analyses are translated into natural languages with techniques, such as annotation and ontology. The other technique for making the outputs perceptible is visualization. At the end of this review, one of the most important issues in the interpretation of omics data analyses is discussed. Omics studies have a large amount of information in their data sets, and every approach reveals only a very restricted aspect of the whole data sets. The understandable messages from these studies have unavoidable uncertainty.

Feature selection with limited bit depth mutual information for portable embedded systems

Since wearable computing systems have grown in importance in the last years, there is an increased interest in implementing machine learning algorithms with reduced precision parameters/computations. Not only learning, also feature selection, most of the times a mandatory preprocessing step in machine learning, is often constrained by the available computational resources. This work considers mutual information – one of the most common measures of dependence used in feature selection algorithms – with a limited number of bits. In order to test the procedure designed, we have implemented it in several well-known feature selection algorithms. Experimental results over several synthetic and real datasets demonstrate that low bit representations are sufficient to achieve performances close to that of double precision parameters and thus open the door for the use of feature selection in embedded platforms that minimize the energy consumption and carbon emissions.

PSBP-SVM: A Machine Learning-Based Computational Identifier for Predicting Polystyrene Binding Peptides

Polystyrene binding peptides (PSBPs) play a key role in the immobilization process. The correct identification of PSBPs is the first step of all related works. In this paper, we proposed a novel support vector machine-based bioinformatic identification model. This model contains four machine learning steps, including feature extraction, feature selection, model training and optimization. In a five-fold cross validation test, this model achieves 90.38, 84.62, 87.50, and 0.90% SN, SP, ACC, and AUC, respectively. The performance of this model outperforms the state-of-the-art identifier in terms of the SN and ACC with a smaller feature set. Furthermore, we constructed a web server that includes the proposed model, which is freely accessible at http://server.malab.cn/PSBP-SVM/index.jsp.

Anatomical Labeling of Human Airway Branches using a Novel Two-Step Machine Learning and Hierarchical Features.

Chronic obstructive pulmonary disease (COPD) is a common inflammatory disease associated with restricted lung airflow. Quantitative computed tomography (CT)-based bronchial measures are popularly used in COPD-related studies, which require both airway segmentation and anatomical branch labeling. This paper presents an algorithm for anatomical labeling of human airway tree branches using a novel two-step machine learning and hierarchical features. Anatomical labeling of airway branches allows standardized spatial referencing of airway phenotypes in large population-based studies. State-of-the-art anatomical labeling methods are associated with mandatory manual reviewing and correction for mislabeled branches-a time-consuming process susceptible to inter-observer variability. The new method is fully automated, and it uses hierarchical branch-level features from the current as well as ancestral and descendant branches. During the first machine learning step, it differentiates candidate anatomical branches from insignificant topological branches, often, responsible for variations in airway branching patterns. The second step is designed for lung lobe-based classification of anatomical labels for valid candidate branches. The machine learning classifiers has been designed, trained, and validated using total lung capacity (TLC) CT scans (n = 350) from the Iowa cohort of the nationwide COPDGene study during their baseline visits. One hundred TLC CT scans were used for training and validation, and a different set of 250 scans were used for testing and evaluative experiments. The new method achieved labeling accuracies of 98.4, 97.2, 92.3, 93.4, and 94.1% in the right upper, right middle, right lower, left upper, and left lower lobe, respectively, and an overall accuracy of 95.9%. For five clinically significant segmental branches, the method has achieved an accuracy of 95.2%.

Multi-label feature selection based on label distribution and feature complementarity

In the real-world, data in various domains usually tend to be high-dimensional, which may result in considerable time complexity and poor performance for multi-label classification problems. Multi-label feature selection is an important preprocessing step in machine learning, which can effectively solve the so-called “curse of dimensionality” by removing irrelevant and redundant features. Nevertheless, the significance of related labels for each instance is generally different, which is an issue that most of the existing multi-label feature selection algorithms have not addressed. Hence, in this paper, we integrate label-distribution learning into multi-label feature selection from the perspective of granular computing with considering multiple feature correlations. Then, a novel multi-label feature selection algorithm based on label distribution and feature complementarity is developed. In addition, the proposed algorithm consists of two primary parts: first, the different significances of related labels for each instance in the multi-label data are obtained based on granular computing; second, the feature complementarity is estimated based on neighborhood mutual information without discretization. Moreover, the superiority of our proposed method over other state-of-the-art methods is demonstrated by conducting comprehensive experiments with 10 publicly available multi-label datasets on six widely-used metrics. Finally, the proposed method can significantly improve the performance of the classifier while reducing the dimension of the original data.

RETRACTED ARTICLE: Automatic epilepsy detection using hybrid decomposition with multi class support vector method

The epilepsy has been detected from the electroencephalogram (EEG) by utilizing the complex wavelet transform with support vector machine. These methods successfully examine each and every frequency in the EEG signal for detecting the epilepsy with effective manner because epilepsy is one of the most important brain abnormalities which affect the entire brain function. The epilepsy is occurring due to the brain stroke, lack of blood flow, brain fever and so on, these lead to create the number of human deaths. So, the brain epilepsy needs to be analyzed and it has to be detected for improving the epilepsy recognition rate. But the major problem with the epilepsy recognition is the accuracy and efficiency of the classifier because of the traditional approximation entropy only extracts the minimum number of features which is difficult to detect the epilepsy with effective manner. These problems increase the false classification rate while analyzing the brain features. So, brain abnormalities has been automatically recognized by using the various machine learning steps like preprocessing, signal decomposition, feature extraction, feature selection and classification. In this research, the brain Epilepsy is recognized by applying the Hybrid Multi Class Support Vector Machine (HMCSVM). Then the performance of the system is analyzed using the experimental results and discussions.

A hybrid feature-selection approach for finding the digital evidence of webapplication attacks

The most critical challenge of web attack forensic investigations is the sheer amount of data and level of complexity. Machine learning technology might be an efficient solution for web attack analysis and investigation. Consequently, machine learning applications have been applied in various areas of information security and digital forensics, and have improved over time. Moreover, feature selection is a crucial step in machine learning; in fact, selecting an optimal feature subset could enhance the accuracy and performance of the predictive model. To date, there has not been an adequate approach to select optimal features for the evidence of web attack. In this study, a hybrid approach that selects the relevant web attack features by combining the filter and wrapper methods is proposed. This approach has been validated by experimental measurements on 3 web attack datasets. The results show that our proposed approach can find the evidence with high recall, high accuracy, and low error rates. We believe that the results presented herein may help us to improve accuracy and recall of machine learning techniques; particularly, in the field of web attack investigation. The tools that use this approach may help digital forensic professionals and law enforcement in finding the evidence much more efficiently and faster.

Prediction of Autism Spectrum Disorder Using Supervised Machine Learning Algorithms

Autism appears to be a neuro developmental disorder that is visible in the early years. It is a wide-spectrum disorder that indicates that the severity and symptoms can vary from person to person. The Centre for Disease Control found that one in 68 was diagnosed with autism spectrum disorder with increasing numbers in every year. Detection of autism in adults is a cumbersome procedure because in adults, many symptoms can blend with some other mental health, motor impairment disorders so misinterpretation of actual diseases can in turn lead to a terrible life without proper diagnosis and effective treatment mechanisms. Machine learning is a powerful computer tool that supports different application domains Learning complex relationships or patterns from large datasets to draw accurate conclusions. Disease assessment can be done with predictive health data analysis and more appropriate treatment mechanisms that are now a hot area of research. Supervised learning is an important step of Machine learning which uses a rule-based approach by examining empirical data sets to build accurate predictive models. In this paper, decision tree, random forest, SVM, neural networks algorithms are applied on autism spectrum data which have been collected from UCI repository. The results of decision tree, random forest, SVM, neural networks algorithms on autism dataset are presented in this paper in an efficient manner. Analysis performed over these accurate results which will be useful to make right decisions in predicting autism spectrum disorder (ASD) at early stages. Thus, early autism intervention using machine learning techniques opens up a new way for autistic individuals to develop the potential to lead a better life by improving their behavioural and emotional skills.

MUSE: Minimum Uncertainty and Sample Elimination Based Binary Feature Selection

This paper presents a novel incremental feature selection method based on minimum uncertainty and feature sample elimination (referred as MUSE). Feature selection is an important step in machine learning. In an incremental feature selection approach, past approaches have attempted to increase class relevance while simultaneously minimizing redundancy with previously selected features. One example of such an approach is the feature selection method of minimum Redundancy Maximum Relevance (mRMR). The proposed approach differs from prior mRMR approach in how the redundancy of the current feature with previously selected features is reduced. In the proposed approach, the feature samples are divided into a pre-specified number of bins; this step is referred to as feature quantization. A novel uncertainty score for each feature is computed by summing the conditional entropies of the bins, and the feature with the lowest uncertainty score is selected. For each bin, its impurity is computed by taking the minimum of the probability of Class 1 and of Class 2. The feature samples corresponding to the bins with impurities below a threshold are discarded and are not used for selection of the subsequent features. The significance of the MUSE feature selection method is demonstrated using the two datasets: arrhythmia and hand digit recognition (Gisette), and datasets for seizure prediction from five dogs and two humans. It is shown that the proposed method outperforms the prior mRMR feature selection method for most cases. For the arrhythmia dataset, the proposed method achieves 30 percent higher sensitivity at the expense of 7 percent loss of specificity. For the Gisette dataset, the proposed method achieves 15 percent higher accuracy for Class 2, at the expense of 3 percent lower accuracy for Class 1. With respect to seizure prediction among 5 dogs and 2 humans, the proposed method achieves higher area-under-curve (AUC) for all subjects.

Active Machine Learning Approach for Crater Detection From Planetary Imagery and Digital Elevation Models

Craters are dominant geomorphological features on the surfaces of the moon, Mars, and other planets. The distribution of craters provides valuable information on the planetary surface geology. Machine learning is a widely used approach to detect craters on planetary surface data. A critical step in machine learning is the determination of training samples. In previous studies, the training samples were mainly selected manually, which usually leads to insufficient numbers due to the high cost and unfavorable quality. Surface imagery and digital elevation models (DEMs) are now commonly available for planetary surfaces; this offers new opportunities for crater detection with better performance. This paper presents a novel active machine learning approach, in which the imagery and DEMs covering the same region are used for collecting training samples with more automation and better performance. In the training process, the approach actively asks for annotations for the 2-D features derived from imagery with inputs from 3-D features derived from the DEMs. Thus, the training pool can be updated accordingly, and the model can be retrained. This process can be conducted several times to obtain training samples in sufficient number and of favorable quality, from which a classifier with better performance can be generated, and it can then be used for automatic crater detection in other regions. The proposed approach highlights two advantages: 1) automatic generation of a large number of high-quality training samples and 2) prioritization of training samples near the classification boundary so as to learn more quickly. Two sets of test data on the moon and Mars were used for the experimental validation. The performance of the proposed approach was superior to that of a regular machine learning method.

On Machine Learning and Knowledge Organization in Multimedia Information Retrieval

Recent technological developments have increased the use of machine learning to solve many problems, including many in information retrieval. Multimedia information retrieval as a problem represents a significant challenge to machine learning as a technological solution, but some problems can still be addressed by using appropriate AI techniques. We review the technological developments and provide a perspective on the use of machine learning in conjunction with knowledge organization to address multimedia IR needs. The semantic gap in multimedia IR remains a significant problem in the field, and solutions to them are many years off. However, new technological developments allow the use of knowledge organization and machine learning in multimedia search systems and services. Specifically, we argue that, the improvement of detection of some classes of low-level features in images music and video can be used in conjunction with knowledge organization to tag or label multimedia content for better retrieval performance. We provide an overview of the use of knowledge organization schemes in machine learning and make recommendations to information professionals on the use of this technology with knowledge organization techniques to solve multimedia IR problems. We introduce a five-step process model that extracts features from multimedia objects (Step 1) from both knowledge organization (Step 1a) and machine learning (Step 1b), merging them together (Step 2) to create an index of those multimedia objects (Step 3). We also overview further steps in creating an application to utilize the multimedia objects (Step 4) and maintaining and updating the database of features on those objects (Step 5).

Large-Scale Automatic Feature Selection for Biomarker Discovery in High-Dimensional OMICs Data

The identification of biomarker signatures in omics molecular profiling is usually performed to predict outcomes in a precision medicine context, such as patient disease susceptibility, diagnosis, prognosis, and treatment response. To identify these signatures, we have developed a biomarker discovery tool, called BioDiscML. From a collection of samples and their associated characteristics, i.e., the biomarkers (e.g., gene expression, protein levels, clinico-pathological data), BioDiscML exploits various feature selection procedures to produce signatures associated to machine learning models that will predict efficiently a specified outcome. To this purpose, BioDiscML uses a large variety of machine learning algorithms to select the best combination of biomarkers for predicting categorical or continuous outcomes from highly unbalanced datasets. The software has been implemented to automate all machine learning steps, including data pre-processing, feature selection, model selection, and performance evaluation. BioDiscML is delivered as a stand-alone program and is available for download at https://github.com/mickaelleclercq/BioDiscML.

LFIT: an unsupervised discretization method based on the Ramer–Douglas–Peucker algorithm

Discretization is the process of converting continuous values into discrete values. It is a preprocessing step of several machine learning and data mining algorithms and the quality of discretization may drastically affect the performance of these algorithms. In this study we propose a discretization algorithm, namely line fitting-based discretization (lFIT), based on the Ramer--Douglas--Peucker algorithm. It is a static, univariate, unsupervised, splitting-based, global, and incremental discretization method where intervals are determined based on the Ramer--Douglas--Peucker algorithm and the quality of partitioning is assessed based on the standard error of the estimate. To evaluate the performance of the proposed method, a set of experiments are conducted on ten benchmark datasets and the achieved results are compared to those obtained by eight state-of-the-art discretization methods. Experimental results show that lFIT achieves higher predictive accuracy and produces less number of inconsistency while it generates larger number of intervals. The obtained results are also validated through Friedman's test and Holm's post hoc test which revealed the fact that lFIT produces discretization schemes that statistically comply both with supervised and unsupervised discretization methods.

A two-stage supervised learning approach for electricity price forecasting by leveraging different data sources

Over the years, the growing penetration of renewable energy into the electricity market has resulted in a significant change in the electricity market price. This change makes the existing forecasting method prone to error, decreasing the economic benefits. Hence, more precise forecasting methods need to be developed. This paper starts with a survey and benchmark of existing machine learning approaches for forecasting the real-time market (RTM) price. While these methods provide sufficient modeling capability via supervised learning, their accuracy is still limited due to the single data source, e.g., historical price information only. In this paper, a novel two-stage supervised learning approach is proposed by diversifying the data sources such as highly correlated power data. This idea is inspired by the recent load forecasting methods that have shown extremely well performances. Specifically, the proposed two-stage method, namely the rerouted method, learns two types of mapping rules. The first one is the mapping between the historical wind power and the historical price. The second is the forecasting rule for wind generation. Based on the two rules, we forecast the price via the forecasted generation and the first learned mapping between power and price. Additionally, we observed that it is not the more training data the better, leading to our validation steps to quantify the best training intervals for different datasets. We conduct comparisons of numerical results between existing methods and the proposed methods based on datasets from the Electric Reliability Council of Texas (ERCOT). For each machine learning step, we examine different learning methods, such as polynomial regression, support vector regression, neural network, and deep neural network. The results show that the proposed method is significantly better than existing approaches when renewables are involved.