Subspace Selection Research Articles

ForesTexter: An efficient random forest algorithm for imbalanced text categorization

In this paper, we propose a new random forest (RF) based ensemble method, ForesTexter, to solve the imbalanced text categorization problems. RF has shown great success in many real-world applications. However, the problem of learning from text data with class imbalance is a relatively new challenge that needs to be addressed. A RF algorithm tends to use a simple random sampling of features in building their decision trees. As a result, it selects many subspaces that contain few, if any, informative features for the minority class. Furthermore, the Gini measure for data splitting is considered to be skew sensitive and bias towards the majority class. Due to the inherent complex characteristics of imbalanced text datasets, learning RF from such data requires new approaches to overcome challenges related to feature subspace selection and cut-point choice while performing node splitting. To this end, we propose a new tree induction method that selects splits, both feature subspace selection and splitting criterion, for RF on imbalanced text data. The key idea is to stratify features into two groups and to generate effective term weighting for the features. One group contains positive features for the minority class and the other one contains the negative features for the majority class. Then, for feature subspace selection, we effectively select features from each group based on the term weights. The advantage of our approach is that each subspace contains adequate informative features for both minority and majority classes. One difference between our proposed tree induction method and the classical RF method is that our method uses Support Vector Machines (SVM) classifier to split the training data into smaller and more balance subsets at each tree node, and then successively retrains the SVM classifiers on the data partitions to refine the model while moving down the tree. In this way, we force the classifiers to learn from refined feature subspaces and data subsets to fit the imbalanced data better. Hence, the tree model becomes more robust for text categorization task with imbalanced dataset. Experimental results on various benchmark imbalanced text datasets (Reuters-21578, Ohsumed, and imbalanced 20 newsgroup) consistently demonstrate the effectiveness of our proposed ForesTexter method. The performance of our proposed approach is competitive against the standard random forest and different variants of SVM algorithms.

Creases and boundary conditions for subdivision curves

Our goal is to find subdivision rules at creases in arbitrary degree subdivision for piece-wise polynomial curves, but without introducing new control points e.g. by knot insertion. Crease rules are well understood for low degree (cubic and lower) curves. We compare three main approaches: knot insertion, ghost points, and modifying subdivision rules. While knot insertion and ghost points work for arbitrary degrees for B-splines, these methods introduce unnecessary (ghost) control points.The situation is not so simple in modifying subdivision rules. Based on subdivision and subspace selection matrices, a novel approach to finding boundary and sharp subdivision rules that generalises to any degree is presented. Our approach leads to new higher-degree polynomial subdivision schemes with crease control without introducing new control points.

Electrode array-based electrical stimulation using ILC with restricted input subspace

Electrode arrays are gaining increasing popularity within the rehabilitation and assistive technology communities, due to their potential to deliver selective electrical stimulation to underlying muscles. This paper develops the first model-based control strategy in this area, unlocking the potential for faster, more accurate postural control. Due to time-varying nonlinear musculoskeletal dynamics, the approach fuses model identification with iterative learning control (ILC), and employs a restricted input subspace comprising only those inputs deemed critical to task completion. The subspace selection embeds past experience and/or structural knowledge, with a dimension chosen to affect a trade-off between the test time and overall accuracy. Experimental results using a 40 element surface electrode array confirm accurate tracking of three reference hand postures.

Large-scale Gauss-Newton inversion of transient controlled-source electromagnetic measurement data using the model reduction framework

Transient data controlled-source electromagnetic measurements are usually interpreted via extracting few frequencies and solving the corresponding inverse frequency-domain problem. Coarse frequency sampling may result in loss of information and affect the quality of interpretation; however, refined sampling increases computational cost. Fitting data directly in the time domain has similar drawbacks, i.e., its large computational cost, in particular, when the Gauss-Newton (GN) algorithm is used for the misfit minimization. That cost is mainly comprised of the multiple solutions of the forward problem and linear algebraic operations using the Jacobian matrix for calculating the GN step. For large-scale 2.5D and 3D problems with multiple sources and receivers, the corresponding cost grows enormously for inversion algorithms using conventional finite-difference time-domain (FDTD) algorithms. A fast 3D forward solver based on the rational Krylov subspace (RKS) reduction algorithm using an optimal subspace selection was proposed earlier to partially mitigate this problem. We applied the same approach to reduce the size of the time-domain Jacobian matrix. The reduced-order model (ROM) is obtained by projecting a discretized large-scale Maxwell system onto an RKS with optimized poles. The RKS expansion replaces the time discretization for forward and inverse problems; however, for the same or better accuracy, its subspace dimension is much smaller than the number of time steps of the conventional FDTD. The crucial new development of this work is the space-time data compression of the ROM forward operator and decomposition of the ROM’s time-domain Jacobian matrix via chain rule, as a product of time- and space-dependent terms, thus effectively decoupling the discretizations in the time and parameter spaces. The developed technique can be equivalently applied to finely sampled frequency-domain data. We tested our approach using synthetic 2.5D examples of hydrocarbon reservoirs in the marine environment.

A Signal Subspace Speech Enhancement Method for Various Noises

In this paper, we propose a single-channel speech enhancement method in terms of subspace techniques to reduce the noises from speech signals in various noises environment. This subspace approach based on Karhunen-Loeve transform and implemented via Principal component analysis. The optimal subspace selection is provided by a minimum description length criterion. An offset factor generated from the white noise was used to modify the variance to adapt to the specified colored noise. Several objective speech quality measures have been introduced to give an overall evaluation of the proposed method. A large amount of data and figures, as well as the audio quality evaluation results, testify that the algorithm provides high performance for the input signal-to-noise ratio range from -5dB to 10dB. It is showed that the proposed approach have excellent characteristics for colored noises in strong background noise environment with lower signal-to-noise ratio. DOI: http://dx.doi.org/10.11591/telkomnika.v11i2.1999

Research on the Speech Enhancement Method based on PCA/KLT Algorithms

In this paper, we propose a speech enhancement technique in terms of subspace methods to reduce the white or colored noise in strong background noise environment. This subspace approach based on Karhunen-Loève transform (KLT) and implemented via Principal Component Analysis (PCA). The subspace selection provided by the minimum description length (MDL) criterion. An offset factor generated from the white noise was used to modify the variance to adapt to the specified colored noise. The objective speech quality measures SegSNR have been introduced to evaluate the performance of the proposed method in time domain. A large amount of data and figures testify that our algorithm provides high performance for a large scale of input signal-to-noise ratio (-5~10dB). The performance of our algorithm is assessed in white and colored noise.

Adaptive Reduced-Rank Estimation of Nonstationary Time-Variant Channels Using Subspace Selection

In this paper, we focus on adaptive time-variant channel estimation for vehicle-to-vehicle (V2V) communications in intelligent transportation systems (ITS) using the IEEE 802.11p physical layer. The IEEE 802.11p pilot pattern is identical to that in the well-known IEEE 802.11a/g (WiFi) standard, which was initially designed for indoor environments with little or no mobility. However, in a V2V drive-by situation, the channel impulse response rapidly changes due to the high relative velocity between transmitter and receiver, as well as the changes in the scattering environment. Hence, for such V2V channels, advanced decision directed channel estimation methods are needed to reach a frame error rate (FER) smaller than 10-1 . Even more importantly, the channels are nonstationary, which implies that the Doppler power spectral density (DSD) and the power delay profile (PDP) change on a timescale comparable with the frame length, which complicates the estimator design. In this paper, we develop an adaptation method for the channel estimation filter that is suitable for the following: 1) the short frame length in IEEE 802.11p; 2) the given pilot structure; and 3) the requirement of only a single received short frame for filter adaptation. We define a set of hypotheses on the support of the DSD and a second set of hypotheses on the support of the PDP. Each hypothesis is represented by a specific subspace spanned by orthogonal basis vectors. For basis vector calculation, we develop a numerically stable algorithm utilizing generalized discrete prolate spheroidal sequences. The adaptation algorithm chooses a hypothesis from both sets such that a probabilistic bound on the channel estimation error is minimized. We implement the hypothesis test by means of a novel subspace selection algorithm that allows utilizing correlated observations of a time- and frequency-selective (2-D) fading process. We validate the adaptive channel estimation scheme in an IEEE 802.11p compliant link level simulation for a relative velocity range from 0 to 111 m/s ≈ 400 km/h ≈ 249 mi/h. Adaptive filtering enables an up to fourfold reduction in the number of required iterations to reach an FER below 10-1 for an Eb/N0 = 12 dB.

Regularized soft K-means for discriminant analysis

Traditionally unsupervised dimensionality reduction methods may not necessarily improve the separability of the data resided in different clusters due to ignorance of the inherent relationship between subspace selection and clustering. It is known that soft clustering using fuzzy c-means or its variants can provide a better and more meaningful data partition than hard clustering, which motivates us to develop a novel entropy regularized soft K-means algorithm for discriminant analysis (ResKmeans) in this paper. ResKmeans performs soft clustering and subspace selection simultaneously and thus gives rise to a generalized linear discriminant analysis (GELDA) which captures both the intra-cluster compactness and the inter-cluster separability. Furthermore, we clarify both the relationship between GELDA and conventional LDA and the inherent relationship between subspace selection and soft clustering. Experimental results on real-world data sets show ResKmeans is superior to other popular clustering algorithms.

Stratified sampling for feature subspace selection in random forests for high dimensional data

For high dimensional data a large portion of features are often not informative of the class of the objects. Random forest algorithms tend to use a simple random sampling of features in building their decision trees and consequently select many subspaces that contain few, if any, informative features. In this paper we propose a stratified sampling method to select the feature subspaces for random forests with high dimensional data. The key idea is to stratify features into two groups. One group will contain strong informative features and the other weak informative features. Then, for feature subspace selection, we randomly select features from each group proportionally. The advantage of stratified sampling is that we can ensure that each subspace contains enough informative features for classification in high dimensional data. Testing on both synthetic data and various real data sets in gene classification, image categorization and face recognition data sets consistently demonstrates the effectiveness of this new method. The performance is shown to better that of state-of-the-art algorithms including SVM, the four variants of random forests (RF, ERT, enrich-RF, and oblique-RF), and nearest neighbor (NN) algorithms.

SNP Selection and Classification of Genome-Wide SNP Data Using Stratified Sampling Random Forests

For high dimensional genome-wide association (GWA) case-control data of complex disease, there are usually a large portion of single-nucleotide polymorphisms (SNPs) that are irrelevant with the disease. A simple random sampling method in random forest using default mtry parameter to choose feature subspace, will select too many subspaces without informative SNPs. Exhaustive searching an optimal mtry is often required in order to include useful and relevant SNPs and get rid of vast of non-informative SNPs. However, it is too time-consuming and not favorable in GWA for high-dimensional data. The main aim of this paper is to propose a stratified sampling method for feature subspace selection to generate decision trees in a random forest for GWA high-dimensional data. Our idea is to design an equal-width discretization scheme for informativeness to divide SNPs into multiple groups. In feature subspace selection, we randomly select the same number of SNPs from each group and combine them to form a subspace to generate a decision tree. The advantage of this stratified sampling procedure can make sure each subspace contains enough useful SNPs, but can avoid a very high computational cost of exhaustive search of an optimal mtry, and maintain the randomness of a random forest. We employ two genome-wide SNP data sets (Parkinson case-control data comprised of 408 803 SNPs and Alzheimer case-control data comprised of 380 157 SNPs) to demonstrate that the proposed stratified sampling method is effective, and it can generate better random forest with higher accuracy and lower error bound than those by Breiman's random forest generation method. For Parkinson data, we also show some interesting genes identified by the method, which may be associated with neurological disorders for further biological investigations.

Boosting k-NN for Categorization of Natural Scenes

The k-nearest neighbors (k-NN) classification rule has proven extremely successful in countless many computer vision applications. For example, image categorization often relies on uniform voting among the nearest prototypes in the space of descriptors. In spite of its good generalization properties and its natural extension to multi-class problems, the classic k-NN rule suffers from high variance when dealing with sparse prototype datasets in high dimensions. A few techniques have been proposed in order to improve k-NN classification, which rely on either deforming the nearest neighborhood relationship by learning a distance function or modifying the input space by means of subspace selection. From the computational standpoint, many methods have been proposed for speeding up nearest neighbor retrieval, both for multidimensional vector spaces and nonvector spaces induced by computationally expensive distance measures. In this paper, we propose a novel boosting approach for generalizing the k-NN rule, by providing a new k-NN boosting algorithm, called UNN (Universal Nearest Neighbors), for the induction of leveraged k-NN. We emphasize that UNN is a formal boosting algorithm in the original boosting terminology. Our approach consists in redefining the voting rule as a strong classifier that linearly combines predictions from the k closest prototypes. Therefore, the k nearest neighbors examples act as weak classifiers and their weights, called leveraging coefficients, are learned by UNN so as to minimize a surrogate risk, which upper bounds the empirical misclassification rate over training data. These leveraging coefficients allows us to distinguish the most relevant prototypes for a given class. Indeed, UNN does not affect the k-nearest neighborhood relationship, but rather acts on top of k-NN search. We carried out experiments comparing UNN to k-NN, support vector machines (SVM) and AdaBoost on categorization of natural scenes, using state-of-the art image descriptors (Gist and Bag-of-Features) on real images from Oliva and Torralba (Int. J. Comput. Vis. 42(3):145–175, 2001), Fei-Fei and Perona (IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 524–531, 2005), and Xiao et al. (IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3485–3492, 2010). Results display the ability of UNN to compete with or beat the other contenders, while achieving comparatively small training and testing times.

Twin support vector machines and subspace learning methods for microcalcification clusters detection

This paper presents a novel framework for microcalcification clusters (MCs) detection in mammograms. The proposed framework has three main parts: (1) first, MCs are enhanced by using a simple-but-effective artifact removal filter and a well-designed high-pass filter; (2) thereafter, subspace learning algorithms can be embedded into this framework for subspace (feature) selection of each image block to be handled; and (3) finally, in the resulted subspaces, the MCs detection procedure is formulated as a supervised learning and classification problem, and in this work, the twin support vector machine (TWSVM) is developed in decision-making of MCs detection. A large number of experiments are carried out to evaluate and compare the MCs detection approaches, and the effectiveness of the proposed framework is well demonstrated.

Classifying Very High-Dimensional Data with Random Forests Built from Small Subspaces

The selection of feature subspaces for growing decision trees is a key step in building random forest models. However, the common approach using randomly sampling a few features in the subspace is not suitable for high dimensional data consisting of thousands of features, because such data often contains many features which are uninformative to classification, and the random sampling often doesn’t include informative features in the selected subspaces. Consequently, classification performance of the random forest model is significantly affected. In this paper, the authors propose an improved random forest method which uses a novel feature weighting method for subspace selection and therefore enhances classification performance over high-dimensional data. A series of experiments on 9 real life high dimensional datasets demonstrated that using a subspace size of features where M is the total number of features in the dataset, our random forest model significantly outperforms existing random forest models.

Feature Extraction Based on Automatic Subspace Partition for Hyperspectral Images

In this paper, a new data subspace partition method is proposed for reduction of hyperspectral image dimensionality. This method involves three steps: subspace partition of whole data space, feature extraction based on principal component analysis (PCA) in subspace and feature selection based on class separability criterion. The main merits of the proposed method are that it much more makes full use of neighboring correlation of hyperspectral data bands than those existing methods, and it realizes automatic subspace partition. In order to testify the effectiveness of the proposed method, classification experiments of hyperspectral images are conducted on AVIRIS data. The experiment investigation shows that the classification result in our new method is improved compared with both segmented principal component transformation (SPCT) and adaptive subspace decomposition method (ASD). When the data dimensionality is reduced 90% by using the proposed method, the overall classification accuracy of nine classes of ground cover reaches 80.2%.

An adaptive‐hybrid meshfree approximation method

SUMMARYIt is now commonly agreed that the global radial basis functions (GRBF) method is an attractive approach for approximating smooth functions. This superiority does not come free; one must find ways to circumvent the associated problem of ill‐conditioning and the high computational cost for solving dense matrix systems. We previously proposed different variants of adaptive methods for selecting proper trial subspaces so that the instability caused by inappropriately shaped parameters were minimized. In contrast, the compactly supported radial basis functions (CSRBF) are more relaxing on the smoothness requirements. By settling with the algebraic order of convergence only, the CSRBF method, provided the support radii are properly chosen, can approximate functions with less smoothness. The reality is that end users must know the functions to be approximated a priori to decide which method to be used; this is not practical if one is solving a time‐evolving partial differential equation. The solution could be smooth at the beginning but the formation of shocks may come later. In this paper, we propose a hybrid algorithm making use of both GRBF and CSRBF with other developed techniques for meshfree approximation with minimal fine tuning. The first contribution here is an adaptive node refinement scheme. Second, we apply the GRBFs (with adaptive subspace selection) on the adaptively generated data sites, and lastly, the CSRBF (with adaptive support selection) that can be used as a blackbox algorithm for robust approximations to a wider class of functions and for solving PDEs. Copyright © 2011 John Wiley & Sons, Ltd.

Improvement of the recursive projection method for linear iterative scheme stabilization based on an approximate eigenvalue problem

An algorithm for stabilizing linear iterative schemes is developed in this study. The recursive projection method is applied in order to stabilize divergent numerical algorithms. A criterion for selecting the divergent subspace of the iteration matrix with an approximate eigenvalue problem is introduced. The performance of the present algorithm is investigated in terms of storage requirements and CPU costs and is compared to the original Krylov criterion. Theoretical results on the divergent subspace selection accuracy are established. The method is then applied to the resolution of the linear advection–diffusion equation and to a sensitivity analysis for a turbulent transonic flow in the context of aerodynamic shape optimization. Numerical experiments demonstrate better robustness and faster convergence properties of the stabilization algorithm with the new criterion based on the approximate eigenvalue problem. This criterion requires only slight additional operations and memory which vanish in the limit of large linear systems.

Subspace Optimization in Centralized Noncoherent MIMO Radar

We consider the problem of subspace optimization for centralized noncoherent multiple input-multiple output (MIMO) radar based on various measures such as capacity, diversity, and probability of detection. In subspace centralized noncoherent MIMO radar (SC-MIMO), a subset of stations is selected based on channel knowledge or channel statistics to reduce system complexity while simultaneously attempting to optimize the performance of the reduced-dimension centralized MIMO radar system. The radar transmitters are assumed to be sufficiently separated (e.g., at different locations) to yield spatially white channel transfer gains and are assumed to operate on a noninterference basis through time-division or frequency-division multiplexing. Detection optimization for the SC-MIMO system in a Neyman Pearson (NP) sense is found to be equivalent to selecting the subspace that maximizes the Frobenius norm of the corresponding channel matrix. Information-theoretic measures for capacity and diversity are also applied to the problem of subspace selection. Channels with temporal coherence times that are long relative to the radar system's latencies and channels with coherence times that are short relative to the radar system's latencies are considered. In the former case, metrics are based upon instantaneous channel estimates, whereas in the latter case, average channel estimates are used. Numerical analyses are conducted to illustrate the use of the metrics for optimizing system performance.

A DOA Estimation Approach under Nonuniform White Noise

The letter deals with direction-of-arrival (DOA) estimation under nonuniform white noise and moderately small signal-to-noise ratios. The proposed approach first uses signal subspace projection for received data vectors, which form an efficient iterative quadratic maximum-likelihood (IQML) approach to achieve fast convergence and high resolution capabilities. In conjunction with a signal subspace selection technique, a more exact signal subspace can be obtained for reducing the nonuniform noise effect. The performance improvement achieved by applying the proposal to the classic IQML method is confirmed by computer simulations.

Distance metric learning guided adaptive subspace semi-supervised clustering

Most existing semi-supervised clustering algorithms are not designed for handling high-dimensional data. On the other hand, semi-supervised dimensionality reduction methods may not necessarily improve the clustering performance, due to the fact that the inherent relationship between subspace selection and clustering is ignored. In order to mitigate the above problems, we present a semi-supervised clustering algorithm using adaptive distance metric learning (SCADM) which performs semi-supervised clustering and distance metric learning simultaneously. SCADM applies the clustering results to learn a distance metric and then projects the data onto a low-dimensional space where the separability of the data is maximized. Experimental results on real-world data sets show that the proposed method can effectively deal with high-dimensional data and provides an appealing clustering performance.

Dimension-splitting data points redistribution for meshless approximation

To better approximate nearly singular functions with meshless methods, we propose a data points redistribution method extended from the well-known one-dimensional equidistribution principle. With properly distributed data points, nearly singular functions can be well approximated by linear combinations of global radial basis functions. The proposed method is coupled with an adaptive trial subspace selection algorithm in order to reduce computational cost. In our numerical examples, clear exponential convergence (with respect to the numbers of data points) can be observed.