Distribution In Feature Space Research Articles

Semi-supervised behavioral learning and its application

Semi-supervised learning has attracted significant attention in pattern recognition and machine learning. Among these methods, a very popular type is semi-supervised support vector machines. However, parameter selection in heat kernel function during the learning process is troublesome and harms the performance improvement of the hypothesis. To solve this problem, a novel local behavioral searching strategy is proposed for semi-supervised learning in this paper. In detail, based on human behavioral learning theory, the support vector machine is regularized with the un-normalized graph Laplacian. After building local distribution of feature space, local behavioral paradigm considers the form of the underlying probability distribution in the neighborhood of a point. Validation of the proposed method is performed with toy and real-life data sets. Results demonstrate that compared with traditional method, our method can more effectively and stably enhance the learning performance.

Domain Invariant Transfer Kernel Learning

Domain transfer learning generalizes a learning model across training data and testing data with different distributions. A general principle to tackle this problem is reducing the distribution difference between training data and testing data such that the generalization error can be bounded. Current methods typically model the sample distributions in input feature space, which depends on nonlinear feature mapping to embody the distribution discrepancy. However, this nonlinear feature space may not be optimal for the kernel-based learning machines. To this end, we propose a transfer kernel learning (TKL) approach to learn a domain-invariant kernel by directly matching source and target distributions in the reproducing kernel Hilbert space (RKHS). Specifically, we design a family of spectral kernels by extrapolating target eigensystem on source samples with Mercer's theorem. The spectral kernel minimizing the approximation error to the ground truth kernel is selected to construct domain-invariant kernel machines. Comprehensive experimental evidence on a large number of text categorization, image classification, and video event recognition datasets verifies the effectiveness and efficiency of the proposed TKL approach over several state-of-the-art methods.

Semi-Supervised Learning by Local Behavioral Searching Strategy

Semi-supervised learning has attracted a significant amount of attention in pattern recognition and machine learning. Among these methods, a very popular type is semi-supervised support vector machines. However, parameter selection in heat kernel function during the learning process is troublesome and harms the performance improvement of the hypothesis. To solve this problem, a novel local behavioral searching strategy is proposed for semi-supervise d learning in this paper. In detail, based on human behavioral learning theory, the support vector machine is regularized with the un-normalized graph Laplacian. After building local distribution of feature space, local behavioral paradigm considers the form of the underlying probability distribution in the neighborhood of a point. Validation of the proposed method is performed with extensive experiments. Results demonstrate that compared with traditional method, our method can more effectively and stably enhance the learning performance.

Large margin aggregation of local estimates for medical image classification.

Medical images typically exhibit complex feature space distributions due to high intra-class variation and inter-class ambiguity. Monolithic classification models are often problematic. In this study, we propose a novel Large Margin Local Estimate (LMLE) method for medical image classification. In the first step, the reference images are subcategorized, and local estimates of the test image are computed based on the reference subcategories. In the second step, the local estimates are fused in a large margin model to derive the similarity level between the test image and the reference images, and the test image is classified accordingly. For evaluation, the LMLE method is applied to classify image patches of different interstitial lung disease (ILD) patterns on high-resolution computed tomography (HRCT) images. We demonstrate promising performance improvement over the state-of-the-art.

Human cognitive paradigm and its application in semi-supervised learning

In many practical data mining applications such as web page classification, unlabeled training examples are readily available but labeled ones are fairly expensive to obtain. Therefore, semi-supervised learning algorithms such as Tri-training have attracted much attention. However, mislabeling the unlabeled data during the learning process is an inevitable problem and harms the performance improvement of the hypothesis. To solve this problem, a novel human cognitive paradigm is constructed for semi-supervised learning in this paper. In detail, based on local distribution of feature space, the majority voting scheme is substituted by an estimation of the probability of sample to belong to a certain class as an efficient strategy for data editing. It considers the form of the underlying probability distribution in the neighborhood of a point to identify and remove the mislabeled data. Validation of the proposed method is performed with extensive experiments. Results demonstrate that compared with Tri-training method, our method can more effectively and stably exploit unlabeled data to enhance the learning performance.

Feature evaluation of radar signal based on aggregation, discreteness and divisibility

Quality of intrapulse feature about radar signals has proved to be a significant foundation to decide whether the signals can be differentiated effectively.For evaluating the quality quantitatively,a method adopting fuzziness and close-degree to evaluate intrapulse feature aggregation and discreteness of signals was proposed in this paper.Space distribution of intrapluse feature about radar signals was analyzed with this method,while intrapulse feature aggregation was evaluated by fuzziness and intrapulse feature discreteness was evaluated by close-degree.And for the overlapping states of feature space distribution,a linear separable measure of intrapluse feature about radar signals was put forward by within-class distance,between-class distance and linear discriminated criterion.The simulation results,based on the experiments of two intrapulse features extracted via time-frequency atom approach about five kinds of radar signal,show that the method and measure proposed in the paper are effective and feasible.It provides a new idea and approach for quantitatively evaluating features of the radar emitter signal.

A Multimedia Retrieval Framework Based on Semi-Supervised Ranking and Relevance Feedback

We present a new framework for multimedia content analysis and retrieval which consists of two independent algorithms. First, we propose a new semi-supervised algorithm called ranking with Local Regression and Global Alignment (LRGA) to learn a robust Laplacian matrix for data ranking. In LRGA, for each data point, a local linear regression model is used to predict the ranking scores of its neighboring points. A unified objective function is then proposed to globally align the local models from all the data points so that an optimal ranking score can be assigned to each data point. Second, we propose a semi-supervised long-term Relevance Feedback (RF) algorithm to refine the multimedia data representation. The proposed long-term RF algorithm utilizes both the multimedia data distribution in multimedia feature space and the history RF information provided by users. A trace ratio optimization problem is then formulated and solved by an efficient algorithm. The algorithms have been applied to several content-based multimedia retrieval applications, including cross-media retrieval, image retrieval, and 3D motion/pose data retrieval. Comprehensive experiments on four data sets have demonstrated its advantages in precision, robustness, scalability, and computational efficiency.

Remote Sensing Image Features Estimating Model Based onRobust Statistical Theory and Its Application

Gaussian Mixture Density Modelling and Decomposition (GMDD) is a hierarchical clustering method based on robust statistical theory. Firstly, GMDD is assumed with a mixture group of Gaussian distribution in feature space, then by optimization algorithm the feature which mostly accord with the assumed distribution is hierarchically extracted from space until all of the features in the space are decomposed to a group of featuring pattern. Compared with conventional statistical clustering methods, GMDD's main outstanding superorities are:(1)Initial number of features does not need to be specified a priori; (2)The proportion of noisy data in the mixture can be large; (3)The parameters estimation of each feature is virtually initial independent; and (4)The variability in the shape and size of the feature densities in the mixture is taken into account. The article presented the model named the GMDD based remote sensing image feature estimation model (GIFEM) , and the model of GA space searching optimization is also presented out.

Methodology Multiclass microarray data classification based on confidence evaluation

Microarray technology is becoming a powerful tool for clinical diagnosis, as it has potential to discover gene expression patterns that are characteristic for a particular disease. To date, this possibility has received much attention in the context of cancer research, especially in tumor classification. However, most published articles have concentrated on the development of binary classification methods while neglected ubiquitous multiclass problems. Unfortunately, only a few multiclass classification approaches have had poor predictive accuracy. In an effort to improve classification accuracy, we developed a novel multiclass microarray data classification method. First, we applied a "one versus rest-support vector machine" to classify the samples. Then the classification confidence of each testing sample was evaluated according to its distribution in feature space and some with poor confidence were extracted. Next, a novel strategy, which we named as "class priority estimation method based on centroid distance", was used to make decisions about categories for those poor confidence samples. This approach was tested on seven benchmark multiclass microarray datasets, with encouraging results, demonstrating effectiveness and feasibility.

A hierarchical naive Bayesian network classifier embedded GMM for textural image

To address several problems in high-dimensional textures feature space and the deficiencies of the single Gaussian distribution for remote sensing data, this paper proposes a hierarchical naive Bayesian network classifier embedded in a Gaussian mixture model for high-dimensional textural image classification. High-dimensional features are grouped by the model on the basis of the correlations between them. In this way, the high-dimensional problem is decomposed into multiple problems of lower dimension. At the same time, for each group of features, a Gaussian mixture model is applied to simulate the data distribution in feature space for land covers, which fits the “original” data distribution better than a single Gaussian model. The Gaussian mixture model is embedded as a child node into a naive Bayesian network, and then the final classification result is obtained within the naive Bayesian network classifier framework. Experimental results for the classification of Landsat ETM+ and QuickBird image textures demonstrated that the classification accuracy of this method is better than that of a traditional Bayesian network classifier and some other classical classifiers. Comparing with the method dealing with original high-dimensional features, it is also more efficiency and effectiveness with fewer demand of sample size and lower time complexity.

Bridging Domains Using World Wide Knowledge for Transfer Learning

A major problem of classification learning is the lack of ground-truth labeled data. It is usually expensive to label new data instances for training a model. To solve this problem, domain adaptation in transfer learning has been proposed to classify target domain data by using some other source domain data, even when the data may have different distributions. However, domain adaptation may not work well when the differences between the source and target domains are large. In this paper, we design a novel transfer learning approach, called BIG (Bridging Information Gap), to effectively extract useful knowledge in a worldwide knowledge base, which is then used to link the source and target domains for improving the classification performance. BIG works when the source and target domains share the same feature space but different underlying data distributions. Using the auxiliary source data, we can extract a ¿bridge¿ that allows cross-domain text classification problems to be solved using standard semisupervised learning algorithms. A major contribution of our work is that with BIG, a large amount of worldwide knowledge can be easily adapted and used for learning in the target domain. We conduct experiments on several real-world cross-domain text classification tasks and demonstrate that our proposed approach can outperform several existing domain adaptation approaches significantly.

Effect of finite sample size on feature selection and classification: A simulation study

The small number of samples available for training and testing is often the limiting factor in finding the most effective features and designing an optimal computer-aided diagnosis (CAD) system. Training on a limited set of samples introduces bias and variance in the performance of a CAD system relative to that trained with an infinite sample size. In this work, the authors conducted a simulation study to evaluate the performances of various combinations of classifiers and feature selection techniques and their dependence on the class distribution, dimensionality, and the training sample size. The understanding of these relationships will facilitate development of effective CAD systems under the constraint of limited available samples. Three feature selection techniques, the stepwise feature selection (SFS), sequential floating forward search (SFFS), and principal component analysis (PCA), and two commonly used classifiers, Fisher's linear discriminant analysis (LDA) and support vector machine (SVM), were investigated. Samples were drawn from multidimensional feature spaces of multivariate Gaussian distributions with equal or unequal covariance matrices and unequal means, and with equal covariance matrices and unequal means estimated from a clinical data set. Classifier performance was quantified by the area under the receiver operating characteristic curve Az. The mean Az values obtained by resubstitution and hold-out methods were evaluated for training sample sizes ranging from 15 to 100 per class. The number of simulated features available for selection was chosen to be 50, 100, and 200. It was found that the relative performance of the different combinations of classifier and feature selection method depends on the feature space distributions, the dimensionality, and the available training sample sizes. The LDA and SVM with radial kernel performed similarly for most of the conditions evaluated in this study, although the SVM classifier showed a slightly higher hold-out performance than LDA for some conditions and vice versa for other conditions. PCA was comparable to or better than SFS and SFFS for LDA at small samples sizes, but inferior for SVM with polynomial kernel. For the class distributions simulated from clinical data, PCA did not show advantages over the other two feature selection methods. Under this condition, the SVM with radial kernel performed better than the LDA when few training samples were available, while LDA performed better when a large number of training samples were available. None of the investigated feature selection-classifier combinations provided consistently superior performance under the studied conditions for different sample sizes and feature space distributions. In general, the SFFS method was comparable to the SFS method while PCA may have an advantage for Gaussian feature spaces with unequal covariance matrices. The performance of the SVM with radial kernel was better than, or comparable to, that of the SVM with polynomial kernel under most conditions studied.

An elliptical basis function network for classification of remote sensing images

An elliptical basis function (EBF) network is employed in this study for the classification of remotely sensed images. Though similar in structure, the EBF network differs from the well-known radial basis function (RBF) network by incorporating full covariance matrices and employing the expectation-maximization (EM) algorithm to estimate the basis functions. Since remotely sensed data often take on mixture-density distributions in the feature space, the network not only possesses the advantage of the RBF mechanism, but also utilizes the EM algorithm to compute the maximum likelihood estimates of the mean vectors and covariance matrices of a Gaussian mixture distribution in the training phase. Experimental results show that the EM-based EBF network is more effective in training and simpler in structure than an RBF network constructed for the same task.

Toward intelligent training of supervised image classifications: directing training data acquisition for SVM classification

Conventional approaches to training a supervised image classification aim to fully describe all of the classes spectrally. To achieve a complete description of each class in feature space, a large training set is typically required. It is not, however, always necessary to have training statistics that provide a complete and representative description of the classes, especially if using nonparametric classifiers. For classification by a support vector machine, only the training samples that are support vectors, which lie on part of the edge of the class distribution in feature space, are required; all other training samples provide no contribution to the classification analysis. If regions likely to furnish support vectors can be identified in advance of the classification, it may be possible to intelligently select useful training samples. The ability to target useful training samples may allow accurate classification from small training sets. This potential for intelligent training sample collection was explored for the classification of agricultural crops from multispectral satellite sensor data. With a conventional approach to training, only a quarter of the training samples acquired actually made a positive contribution to the analysis and allowed the crops to be classified to a high accuracy (92.5%). The majority of the training set, therefore, was unnecessary as it made no contribution to the analysis. Using ancillary information on soil type, however, it would be possible to constrain the training sample acquisition process. By limiting training sample acquisition only to regions with a specific soil type, it was possible to use a small training set to classify the data without loss of accuracy. Thus, a small number of intelligently selected training samples may be used to classify a data set as accurately as a larger training set derived in a conventional manner. The results illustrate the potential to direct training data acquisition strategies to target the most useful training samples to allow efficient and accurate image classification.

KNOWLEDGE-GUIDED CLASSIFICATION OF COASTAL ZONE COLOR IMAGES OFF THE WEST FLORIDA SHELF

A knowledge-guided approach to automatic classification of Coastal Zone Color images off the West Florida Shelf is described. The approach is used to identify red tides on the West Florida Shelf, as well as areas with high concentration of dissolved organic matter such as a river plume found seasonally along the West Florida coast over the middle of the shelf. The Coastal Zone Color images are initially segmented by the unsupervised Multistage Random Sampling Fuzzy c-Means algorithm. Then, a knowledge-guided system is applied to the centroid values of resultant clusters to label case I, case II waters, a dilute river plume ("green river"), and red tide. The domain knowledge base contains information on cluster distribution in feature space, as well as spatial information such as bathymetry data. Our knowledge base consists of a rule-guided system and an embedded neural network. From 60 images, after training the system, this procedure recognizes all 15 images which contained a river plume and 45 images without. The system can correctly classify 74% of the pixels that belong to the river plume, which provides a substantial advantage to users looking for offshore extensions of riverine influence. Red tides are also successfully identified in a time series of images for which ground truth confirmed the presence of a harmful bloom.

Generalization properties of finite-size polynomial support vector machines

The learning properties of finite-size polynomial support vector machines are analyzed in the case of realizable classification tasks. The normalization of the high-order features acts as a squeezing factor, introducing a strong anisotropy in the patterns distribution in feature space. As a function of the training set size, the corresponding generalization error presents a crossover, more or less abrupt depending on the distribution's anisotropy and on the task to be learned, between a fast-decreasing and a slowly decreasing regime. This behavior corresponds to the stepwise decrease found by Dietrich et al. [Phys. Rev. Lett. 82, 2975 (1999)] in the thermodynamic limit. The theoretical results are in excellent agreement with the numerical simulations.

On the Distribution and Convergence of Feature Space in Self-Organizing Maps

In this paper an analysis of the statistical and the convergence properties of Kohonen's self-organizing map of any dimension is presented. Every feature in the map is considered as a sum of a number of random variables. We extend the Central Limit Theorem to a particular case, which is then applied to prove that the feature space during learning tends to multiple gaussian distributed stochastic processes, which will eventually converge in the mean-square sense to the probabilistic centers of input subsets to form a quantization mapping with a minimum mean squared distortion either globally or locally. The diminishing effect, as training progresses, of the initial states on the value of the feature map is also shown.

Knowledge-based classification and tissue labeling of MR images of human brain

Presents a knowledge-based approach to automatic classification and tissue labeling of 2D magnetic resonance (MR) images of the human brain. The system consists of 2 components: an unsupervised clustering algorithm and an expert system. MR brain data is initially segmented by the unsupervised algorithm, then the expert system locates a landmark tissue or cluster and analyzes it by matching it with a model or searching in it for an expected feature. The landmark tissue location and its analysis are repeated until a tumor is found or all tissues are labeled. The knowledge base contains information on cluster distribution in feature space and tissue models. Since tissue shapes are irregular, their models and matching are specially designed: 1) qualitative tissue models are defined for brain tissues such as white matter; 2) default reasoning is used to match a model with an MR image; that is, if there is no mismatch between a model and an image, they are taken as matched. The system has been tested with 53 slices of MR images acquired at different times by 2 different scanners. It accurately identifies abnormal slices and provides a partial labeling of the tissues. It provides an accurate complete labeling of all normal tissues in the absence of large amounts of data nonuniformity, as verified by radiologists. Thus the system can be used to provide automatic screening of slices for abnormality. It also provides a first step toward the complete description of abnormal images for use in automatic tumor volume determination.