Problem Of Dimensionality Reduction Research Articles

Dual Random Projection for Linear Support Vector Machine

Support Vector Machine (SVM) is a popular machine learning methods in the field of data analysis. The Random Projections (RP) method can solve the problem of dimensionality reduction of high-dimensional data quickly and effectively to reduce the computational cost of the related optimization problem and is widely used in the SVM method. However, the large-scale SVM has the problem of reduced classification accuracy after dimension reduction by random projection feature. Therefore, we propose a linear kernel support vector machine based on dual-random projection (drp-LSVM) for large-scale classification problem, which combines the duality recovery theory. In this paper, we analyze the geometric properties of drp-LSVM, and prove that while dividing the geometric advantage of support vector machine based on random projection (rp-LSVM), the division of hyperplane is closer to that obtained by all data training Original classifier. The experimental results show that the drp-LSVM can reduce the classification error and improve the training precision, and the performance evaluation is closer to the classifier trained with the original data while inheriting the advantages of rp-LSVM.

Computer geometry algorithms in feature space dimension reduction problems

One of the goals of multidimensional feature space dimension reduction is to visualize the arrangement of classes on a plane. The quality of such a representation depends on the dimension-reduction method and the used measure of the closeness of classes. In this paper, we propose a technology of dimension reduction to two dimensions based on the calculation of convex hulls of classes and determination of their intersection degree. Experiments carried out in the MATLAB system show that this technology on the training sample makes it possible to identify the minimum potentially achievable intersection degree of classes intersecting in a multidimensional space and to obtain a mutual mapping of classes on the plane corresponding to the minimum degree of their intersection.

Direct Estimation of the Derivative of Quadratic Mutual Information with Application in Supervised Dimension Reduction.

A typical goal of linear-supervised dimension reduction is to find a low-dimensional subspace of the input space such that the projected input variables preserve maximal information about the output variables. The dependence-maximization approach solves the supervised dimension-reduction problem through maximizing a statistical dependence between projected input variables and output variables. A well-known statistical dependence measure is mutual information (MI), which is based on the Kullback-Leibler (KL) divergence. However, it is known that the KL divergence is sensitive to outliers. Quadratic MI (QMI) is a variant of MI based on the [Formula: see text] distance, which is more robust against outliers than the KL divergence, and a computationally efficient method to estimate QMI from data, least squares QMI (LSQMI), has been proposed recently. For these reasons, developing a supervised dimension-reduction method based on LSQMI seems promising. However, not QMI itself but the derivative of QMI is needed for subspace search in linear-supervised dimension reduction, and the derivative of an accurate QMI estimator is not necessarily a good estimator of the derivative of QMI. In this letter, we propose to directly estimate the derivative of QMI without estimating QMI itself. We show that the direct estimation of the derivative of QMI is more accurate than the derivative of the estimated QMI. Finally, we develop a linear-supervised dimension-reduction algorithm that efficiently uses the proposed derivative estimator and demonstrate through experiments that the proposed method is more robust against outliers than existing methods.

Nonlinear sufficient dimension reduction for functional data

We propose a general theory and the estimation procedures for nonlinear sufficient dimension reduction where both the predictor and the response may be random functions. The relation between the response and predictor can be arbitrary and the sets of observed time points can vary from subject to subject. The functional and nonlinear nature of the problem leads to construction of two functional spaces: the first representing the functional data, assumed to be a Hilbert space, and the second characterizing nonlinearity, assumed to be a reproducing kernel Hilbert space. A particularly attractive feature of our construction is that the two spaces are nested, in the sense that the kernel for the second space is determined by the inner product of the first. We propose two estimators for this general dimension reduction problem, and establish the consistency and convergence rate for one of them. These asymptotic results are flexible enough to accommodate both fully and partially observed functional data. We investigate the performances of our estimators by simulations, and applied them to data sets about speech recognition and handwritten symbols.

Ant colony optimization with partial-complete searching for attribute reduction

The time-cost-sensitive attribute reduction problem is more challenging than the classical reduct problem since the optimal solution is sparser. Ant colony optimization (ACO) is an effective approach to this problem. However, the efficiency is unsatisfactory since each ant needs to search for a complete solution. In this paper, we propose a partial-complete searching technique for ACO and design the APC algorithm. Partial searching is undertaken by pioneer ants through selecting only a few attributes to save time, while complete searching is undertaken by harvester ants for complete solutions. Experiments are undertaken on seven real-world and a set of artificial datasets with various settings of costs. Compared with two bio-inspired and two greedy algorithms, APC is more efficient while obtaining the same level of quality metrics. The APC algorithm can be also extended for other combinatorial optimization problems.

Linear discriminant analysis: A detailed tutorial

Linear Discriminant Analysis (LDA) is a very common technique for dimensionality reduction problems as a preprocessing step for machine learning and pattern classification applications. At the same time, it is usually used as a black box, but (sometimes) not well understood. The aim of this paper is to build a solid intuition for what is LDA, and how LDA works, thus enabling readers of all levels be able to get a better understanding of the LDA and to know how to apply this technique in different applications. The paper first gave the basic definitions and steps of how LDA technique works supported with visual explanations of these steps. Moreover, the two methods of computing the LDA space, i.e. class-dependent and class-independent methods, were explained in details. Then, in a step-by-step approach, two numerical examples are demonstrated to show how the LDA space can be calculated in case of the class-dependent and class-independent methods. Furthermore, two of the most common LDA problems (i.e. Small Sample Size (SSS) and non-linearity problems) were highlighted and illustrated, and state-of-the-art solutions to these problems were investigated and explained. Finally, a number of experiments was conducted with different datasets to (1) investigate the effect of the eigenvectors that used in the LDA space on the robustness of the extracted feature for the classification accuracy, and (2) to show when the SSS problem occurs and how it can be addressed.

Compressive sensing-based feature extraction for bearing fault diagnosis using a heuristic neural network

Bearing fault diagnosis collects massive amounts of vibration data about a rotating machinery system, whose fault classification largely depends on feature extraction. Features reflecting bearing work states are directly extracted using time-frequency analysis of vibration signals, which leads to high dimensional feature data. To address the problem of feature dimension reduction, a compressive sensing-based feature extraction algorithm is developed to construct a concise fault feature set. Next, a heuristic PSO-BP neural network, whose learning process perfectly combines particle swarm optimization and the Levenberg–Marquardt algorithm, is constructed for fault classification. Numerical simulation experiments are conducted on four datasets sampled under different severity levels and load conditions, which verify that the proposed fault diagnosis method achieves efficient feature extraction and high classification accuracy.

Low-rank preserving embedding

In this paper, we consider the problem of linear dimensionality reduction with the novel technique of low-rank representation, which is a promising tool of discovering subspace structures of given data. Existing approaches based on graph embedding usually capture structure of data via stacking the local structure of each datum, such as neighborhood graph, ℓ1-graph and ℓ2-graph. Yet they lack explicit discrimination between those local structures and suffer from corrupted samples. To this end, we propose a new linear dimensionality reduction method by virtue of the lowest rank representation (LRR) of data, which is dubbed low-rank preserving embedding (LRPE). Different from the traditional routes, LRPE achieves all data self-representations jointly and can thus extract the global structure of a data set as a whole. The global low-rank constraint explicitly enforces the LRR matrix to be block-diagonal form, so that the samples with a similar intrinsic structure, which are more likely to be from the same class, are described by a similar set of bases. Hence, LRPE is discriminative even if no class labels are provided. Benefiting from the robust LRR, LRPE is also robust to various noises and errors in data. Besides, we rewritten all related methods into a unified formulation, followed by a detailed solution and clear comparisons. Finally, we conduct extensive experiments on publicly available data sets for data visualization and classification. The inspiring experimental results show the effectiveness, the cheap computation and the robustness of the proposed method.

Feature Selection Based on High Dimensional Model Representation for Hyperspectral Images.

In hyperspectral image analysis, the classification task has generally been addressed jointly with dimensionality reduction due to both the high correlation between the spectral features and the noise present in spectral bands, which might significantly degrade classification performance. In supervised classification, limited training instances in proportion with the number of spectral features have negative impacts on the classification accuracy, which is known as Hughes effects or curse of dimensionality in the literature. In this paper, we focus on dimensionality reduction problem, and propose a novel feature-selection algorithm, which is based on the method called high dimensional model representation. The proposed algorithm is tested on some toy examples and hyperspectral datasets in comparison with conventional feature-selection algorithms in terms of classification accuracy, stability of the selected features and computational time. The results show that the proposed approach provides both high classification accuracy and robust features with a satisfactory computational time.

Attribute reduction for sequential three-way decisions under dynamic granulation

In real-world decision making, sequential three-way decisions are an effective way of human problem solving under multiple levels of granularity. Making the right decision at the most optimal level is a crucial issue. To this end, we address the attribute reduction problem for sequential three-way decisions under dynamic granulation. By reviewing the existing definitions of attribute reducts, a new attribute reduct for sequential three-way decisions is defined, and a corresponding monotonic attribute significance measure is designed. An attribute reduction algorithm satisfying the monotonicity of the probabilistic positive region is developed. The relationships of the different attribute reducts, the probabilistic positive regions and the probabilistic positive rules for decision-theoretic rough set models are further discussed under global view, local view and sequential three-way decisions. Experimental results demonstrate that our method is effective. This study will provide a new insight into the attribute reduction problem of sequential three-way decisions.

A new web-based solution for modelling data mining processes

The conventional technologies and methods are not able to store and analyse recent data that come from different sources: various devices, sensors, networks, transactional applications, the web, and social media. Due to a complexity of data, data mining methods should be implemented using the capabilities of the Cloud technologies. In this paper, a new web-based solution named DAMIS, inspired by the Cloud, is proposed and implemented. It allows making massive data mining simpler, effective, and easily understandable for data scientists and business intelligence professionals by constructing scientific workflows for data mining using a drag and drop interface. The usage of scientific workflows allows composing convenient tools for modelling data mining processes and for simulation of real-world time- and resource-consuming data mining problems. The solution is useful to solve data classification, clustering, and dimensionality reduction problems. The DAMIS architecture is designed to ensure easy accessibility, usability, scalability, and portability of the solution. The proposed solution has a wide range of applications and allows to get deep insights into the data during the process of knowledge discovery.

THE COMPLEX DATA DIMENSIONALITY REDUCTION FOR DIAGNOSTIC AND RECOGNITION MODEL BUILDING ON PRECEDENTS

The problem of data dimensionality reduction for diagnostic and recognizing model construction is solved. The object of study is the process of data-driven diagnosis. The subject of study is the data reduction methods for diagnostic model construction on precedents. The purpose of work is to create a set of indicators to quantify the importance of instances and features, as well as a method of data sample dimensionality reduction in the diagnosis and pattern recognition and problem solving. The mathematical support for the sample formation and feature selection is developed on the base of common approach to the assessment of their significance. The set of indicators is proposed to quantify the individual informativity of instances and features in the local neighborhood in the feature space. The exhaustive search methods for data sample dimensionality reduction in the solution of recognition and diagnosis problems have been further developed. They are modified by taking into account of the offered individual estimations of informativity of instances and features in the search operators. The proposed methods and indicator complex are implemented as software and studied in the solution of data dimensionality reduction problems. The conducted experiments confirmed the efficiency of the developed mathematical tools and allow to recommend them for use in practice for solving the problems of non-destructive diagnosis and pattern recognition on features.

An improved social spider optimization algorithm based on rough sets for solving minimum number attribute reduction problem

The minimum number attribute reduction problem is an important issue when dealing with huge amounts of data. The problem of minimum attribute reduction is formally known to be as an NP complete nonlinearly constrained optimization problem. Social spider optimization algorithm is a new meta-heuristic algorithm of the swarm intelligence field to global solution. The social spider optimization algorithm is emulates the behavior of cooperation between spiders based on the biological laws of the cooperative colony. Inspired by the social spiders, in this paper, an improved social spider algorithm for the minimal reduction problem was proposed. In the proposed algorithm, the fitness function depends on the rough sets dependency degree and it takes into a consideration the number of selected features. For each spider, the fitness function is computed and compared with the global best fitness value. If the current value is better, then the global best fitness is replaced with it and its position became the reduct set. Then, the position of each spider is updated according to its type. This process is repeated until the stopping criterion is satisfied. To validate the proposed algorithm, several real clinical medical datasets which are available from the UCI Machine Learning Repository were used to compute the performance of the proposed algorithm. The experimental results illustrate that the proposed algorithm is superior to state-of-the-art swarm-based in terms of classification accuracy while limiting number of features.

Dimension reduction for microarray data using multi-objective ant colony optimisation

DNA microarray technology is a nascent technology having vast applications in the molecular biology. Despite of its many useful applications in disease diagnosis and drug discovery, analysing DNA microarray data has become a challenge for bio-analysts. Microarrays present curse of dimensionality problem and call for the development of new techniques to handle it. The problem of dimensionality reduction or feature/gene selection has been posed as a multi-objective problem in the literature and can be better solved using multi-objective meta-heuristics. In this paper, we have proposed a multi-objective ant colony optimisation (MOACO) algorithm for gene selection. The contribution of this paper is to obtain multiple non-dominated solutions instead of a single best solution. These multiple solutions enable a user to select a solution according to his/her preference or application domain. The predicted genes also called as bio-markers, can help in disease diagnosis and may direct the progress of drug efficacy. A comparative analysis of the proposed method has been done using some conventional feature selection techniques and a method based on multi-objective particle swarm optimisation (MOPSO) proposed by Mukhopadhyay and Mandal (2014). The results obtained confirm the superiority of MOACO-based approach over the others in terms of various performance metrics.

Dimension reduction for microarray data using multi-objective ant colony optimisation

DNA microarray technology is a nascent technology having vast applications in the molecular biology. Despite of its many useful applications in disease diagnosis and drug discovery, analysing DNA microarray data has become a challenge for bio-analysts. Microarrays present curse of dimensionality problem and call for the development of new techniques to handle it. The problem of dimensionality reduction or feature/gene selection has been posed as a multi-objective problem in the literature and can be better solved using multi-objective meta-heuristics. In this paper, we have proposed a multi-objective ant colony optimisation (MOACO) algorithm for gene selection. The contribution of this paper is to obtain multiple non-dominated solutions instead of a single best solution. These multiple solutions enable a user to select a solution according to his/her preference or application domain. The predicted genes also called as bio-markers, can help in disease diagnosis and may direct the progress of drug efficacy. A comparative analysis of the proposed method has been done using some conventional feature selection techniques and a method based on multi-objective particle swarm optimisation (MOPSO) proposed by Mukhopadhyay and Mandal (2014). The results obtained confirm the superiority of MOACO-based approach over the others in terms of various performance metrics.

Solving feature selection problem using intelligent double treatment iterative composite neighbourhood structure algorithm

Attribute reduction is one of the main contributions in rough set theory (RST) that tries to discover all possible reducts by eliminating redundant attributes while maintaining the information of the problem in hand. In this paper, we propose a meta-heuristic methodology called a double treatment iterative improvement algorithm with intelligent selection of composite neighbourhood structure, to solve the attribute reduction problems and to obtain near optimal reducts. The algorithm works iteratively by only accepting an improved solution. The proposed approach has been tried on a set of 13 benchmark datasets taken from the University of California Irvine (UCI) machine learning repository in line with the state-of-the-art methods. Thirteen datasets have been chosen due to the differences in size and complexity in order to test the stability of the proposed algorithm. The experimental results demonstrate that the proposed approach is able to produce competitive results for the tested datasets.

Solving feature selection problem using intelligent double treatment iterative composite neighbourhood structure algorithm

Attribute reduction is one of the main contributions in rough set theory (RST) that tries to discover all possible reducts by eliminating redundant attributes while maintaining the information of the problem in hand. In this paper, we propose a meta-heuristic methodology called a double treatment iterative improvement algorithm with intelligent selection of composite neighbourhood structure, to solve the attribute reduction problems and to obtain near optimal reducts. The algorithm works iteratively by only accepting an improved solution. The proposed approach has been tried on a set of 13 benchmark datasets taken from the University of California Irvine (UCI) machine learning repository in line with the state-of-the-art methods. Thirteen datasets have been chosen due to the differences in size and complexity in order to test the stability of the proposed algorithm. The experimental results demonstrate that the proposed approach is able to produce competitive results for the tested datasets.

Robust Locality Preserving Projections With Cosine-Based Dissimilarity for Linear Dimensionality Reduction

Locality preserving projection (LPP) is a classical tool for dimensionality reduction problems. However, it is sensitive to outliers because of utilizing the $\ell _{2}$ -norm-based distance criterion. In this paper, we propose a new approach, termed Euler-LPP, by preserving the local structures of data under the distance criterion of the cosine-based dissimilarity. Euler-LPP is robust to outliers in that the cosine-based dissimilarity suppresses the influence of outliers more efficiently than the $\ell _{2}$ -norm. An explicit mapping, defined by a complex kernel (euler kernel) is adopted to map the data from the input space to complex reproducing kernel Hilbert spaces (CRKHSs), in which the distance of the data pairs under the $\ell _{2}$ -norm is equal to that in the input space under the cosine-based dissimilarity. Thus, the robust dimensionality problem can be directly solved in CRKHS, where the solution is guaranteed to converge to a global minimum. In addition, Euler-LPP is easy to implement without significantly increasing computational complexity. Experiment results on several benchmark databases confirm the effectiveness of the proposed method.

About a Fuzzy Distance between Two Fuzzy Partitions and Application in Attribute Reduction Problem

Abstract According to traditional rough set theory approach, attribute reduction methods are performed on the decision tables with the discretized value domain, which are decision tables obtained by discretized data methods. In recent years, researches have proposed methods based on fuzzy rough set approach to solve the problem of attribute reduction in decision tables with numerical value domain. In this paper, we proposeafuzzy distance between two partitions and an attribute reduction method in numerical decision tables based on proposed fuzzy distance. Experiments on data sets show that the classification accuracy of proposed method is more efficient than the ones based fuzzy entropy.

A general reduction algorithm for relation decision systems and its applications

This paper studies the attribute reduction problem for general relation decision systems. We propose a new discernibility matrix to solve this problem. Combining the discernibility matrix and a recently proposed fast algorithm, we propose a simple and unified attribute reduction algorithm for relation decision systems that is not contingent on the consistency of relation decision systems. We derive the reduction algorithm for the special cases of complete, incomplete, and numerical decision tables. As an application, we transform the attribute reduction of relation decision systems into one for covering decision systems. This gives a convenient and effective reduction algorithm for covering decision systems. The reduction results obtained using University of California Irvine data sets show that the proposed algorithm is simple and efficient. Moreover, the proposed algorithm enables the results of classical attribute reduction approaches to be reinterpreted, giving them far greater unification and generality.