Prototype Selection Methods Research Articles

A nearest neighbor multiple-point statistics method for fast geological modeling

Multiple-point statistics (MPS) is a powerful method to generate realistic geological models. Given a training image as a prior model, the program iteratively reproduces spatial patterns in the simulation grid. However, running speed becomes a limitation to practical applications. MPS has to handle complicated and high-dimensional structures at the cost of simulation time. With the objective to accelerate geostatistical modeling with categorical variables, we propose a nearest neighbor simulation (NNSIM) method. Several k-nearest neighbor (kNN) classifiers are incorporated into MPS framework. First, we identify representative patterns with a prototype selection method. Different from existing MPS programs, our method selects training patterns according to their influences on simulation quality. A pattern subset of small size has a positive effect on searching time. Second, a teacher-student architecture is suggested to improve the pattern subset. In order to address missing data, our program augments the subset with key patterns during simulation. A cosine distance metric is applied to compare the original dataset and pattern subset. Third, our program organizes patterns with a ball tree. Pattern groups with low similarity are dynamically removed to fulfill fast search. We examine the proposed NNSIM by a benchmark channel simulation, a 2D flume model, and a 3D sandstone modeling. Many quantitative approaches are employed to evaluate geometrical and physical properties. The experimental results indicate that our NNSIM significantly improves the computational efficiency while exhibits comparable simulation quality to traditional MPS programs.

K-nearest neighbors rule combining prototype selection and local feature weighting for classification

K-Nearest Neighbors (KNN) rule is a simple yet powerful classification technique in machine learning. Nevertheless, it suffers from some drawbacks such as high memory consumption, low time efficiency, class overlapping and difficulty of setting an appropriate K value. In this study, we propose an Improved K-Nearest Neighbor rule combining Prototype Selection and Local Feature Weighting (IKNN_PSLFW) to address the above issues in one framework. Differing from conventional prototype selection, IKNN_PSLFW not only selects the representative instances as prototypes, but also preserves the information of instances that are not selected. To deal with the class overlapping problem, IKNN_PSLFW explores the feature relevance in local regions by assigning different weights to different features. For an instance with unknown class label, IKNN_PSLFW uses three classification rules corresponding to three scenarios, according to the distance between the instance and each prototype, for classification. To evaluate the performance of IKNN_PSLFW, we conduct experimental study on 20 benchmark datasets. The experimental results show that compared with the conventional KNN rule, some state-of-the-art prototype selection methods and other machine learning algorithms, the proposed IKNN_PSLFW achieves promising classification performance with high time efficiency.

Quad division prototype selection-based k-nearest neighbor classifier for click fraud detection from highly skewed user click dataset

In online advertising, the user-clicks dataset based fraudulent publishers’ classification models exhibit poor performance due to high skewness in class distribution of the publishers. The nearest-neighbor based classification techniques are popularly used to reduce the impact of class skewness on performance. The Nearest-Neighbor techniques use Prototype Selection (PS) methods to select promising samples before classifying them for reducing the size of training data. Although Nearest-Neighbor techniques are simple to use and reduce the negative impact of the loss of potential information, they suffer from higher storage requirements and slower classification speed when applied on datasets with skewed class distributions. In this paper, we propose a Quad Division Prototype Selection-based k-Nearest Neighbor classifier (QDPSKNN) by introducing quad division method for handling uneven class distribution. The quad-division divides the data into four quartiles (groups) and performs controlled under-sampling for balancing class distribution. It reduces the size of the training dataset by selecting only the relevant prototypes in the form of nearest-neighbors. The performance of QDPSKNN is evaluated on Fraud Detection in Mobile Advertising (FDMA) user-click dataset and fifteen other benchmark imbalanced datasets to test its generalizing behaviour. The performance is also compared with one baseline model (k-NN) and four other prototype selection methods such as NearMiss-1, NearMiss-2, NearMiss-3, and Condensed Nearest-Neighbor. The results show improved classification performance with QDPSKNN in terms of precision, recall, f-measure, g-mean, reduction rate and execution time, compared to existing prototype selection methods in the classification of fraudulent publishers as well as on other benchmark imbalanced datasets. Wilcoxon signed ranked test is conducted to demonstrate significant differences amid QDPSKNN and state-of-the-art methods.

Prototype Selection Method Based on the Rivality and Reliability Indexes for the Improvement of the Classification Models and External Predictions.

Prototype or instance selection techniques is an important field of research in knowledge discovery, data mining, and machine learning. In QSAR, the use of prototype selection techniques in the preprocessing stage of the construction of the QSAR models favors the data set curation, improving the interpretability and accuracy of the models as well as the performance of the algorithms. In this paper, we propose an efficient method for prototype selection to be used in the preprocessing stage of the construction of QSAR classification models. The proposed method is able to generate very high reduction rates in the cardinality of the training set while maintaining or even increasing the accuracy of the classification models. The validation of the method has been carried out by means of the prediction of external molecules, demonstrating that the prediction of new molecules is also maintained or even improved. The method has been tested using 40 benchmark data sets of different sizes and balancing ratios; the results of the tests have demonstrated the wide applicability domain of the proposed method.

Adaptive geometric median prototype selection method for k-nearest neighbors classification

Adaptive geometric median prototype selection method for k-nearest neighbors classification

Time series classification using local distance-based features in multi-modal fusion networks

We propose the use of a novel feature, called local distance features, for time series classification. The local distance features are extracted using Dynamic Time Warping (DTW) and classified using Convolutional Neural Networks (CNN). DTW is classically as a robust distance measure for distance-based time series recognition methods. However, by using DTW strictly as a global distance measure, information about the matching is discarded. We show that this information can further be used as supplementary input information in temporal CNNs. This is done by using both the raw data and the features extracted from DTW in multi-modal fusion CNNs. Furthermore, we explore the effects of different prototype selection methods, prototype numbers, and data fusion schemes induce on the accuracy. We perform experiments on a wide range of time series datasets including three Unipen handwriting datasets, four UCI Machine Learning Repository datasets, and 85 UCR Time Series Classification Archive datasets.

A Neighbor Prototype Selection Method Based on CCHPSO for Intrusion Detection

Nearest neighbor (NN) models play an important role in the intrusion detection system (IDS). However, with the advent of the era of big data, the NN model has the disadvantages of low efficiency, noise sensitivity, and high storage requirement. This paper presents a neighbor prototype selection method based on CCHPSO for intrusion detection. In the model, the prototype selection and feature weight adjustment are performed simultaneously and k-nearest neighbor (KNN) is used as the basic classifier. To deal with large-scale optimization problems, a cooperative coevolving algorithm based on hybrid standard particle swarm and binary particle swarm optimization, which employs the divide-and-conquer strategy, is proposed in this paper. Meanwhile, a fitness function based on the accuracy and data reduction rate is defined in the CCHPSO to obtain a set of appropriate prototypes and feature weights. The KDD99 and NSL datasets are used to assess the effectiveness of the method. The empirical results indicate that the data reduction rate of the proposed method is very high, ranging from 82.32% to 92.01%. Compared with all the data used, the proposed method can not only achieve comparable accuracy performance but also save a lot of storage and computing resources.

Pruning strategies for nearest neighbor competence preservation learners

Abstract In order to alleviate both the spatial and temporal cost of the nearest neighbor classification rule, competence preservation techniques aim at substituting the training set with a selected subset, known as consistent subset. In order to improve generalization and to prevent induction of overly complex models, in this study the application of the Pessimistic Error Estimate (PEE) principle in the context of the nearest neighbor rule is investigated. Generalization is estimated as a trade-off between training set accuracy and model complexity. As major results, it is shown that PEE-like selection strategies guarantee to preserve the accuracy of the consistent subset with a far larger reduction factor and, moreover, that sensible generalization improvements can be obtained by using a reduced subset. Moreover, comparison with state-of-the-art hybrid prototype selection methods highlight that the here introduced FCNN-PAC strategy is able to obtain a model of size comparable to that obtained by the best prototype selection methods, with far smaller time requirements, corresponding to four orders of magnitude on medium-sized datasets.

Nearest Neighbor Based Prototype Classification Preserving Class Regions

A prototype selection method chooses a small set of training points from a whole set of class data. As the data size increases, the selected prototypes play a significant role in covering class regions and learning a discriminate rule. This paper discusses the methods for selecting prototypes in a classification framework. We formulate a prototype selection problem into a set covering optimization problem in which the sets are composed with distance metric and predefined classes. The formulation of our problem makes us draw attention only to prototypes per class, not considering the other class points. A training point becomes a prototype by checking the number of neighbors and whether it is preselected. In this setting, we propose a greedy algorithm which chooses the most relevant points for preserving the class dominant regions. The proposed method is simple to implement, does not have parameters to adapt, and achieves better or comparable results on both artificial and real-world problems.

Selection of effective training instances for scalable automatic image annotation

Automatic image annotation means employing learning models for describing visual contents of images by using text descriptors. With the fast growth of digital images in the web, large-scale automatic image annotation has started to deal with major challenges. The most important challenges are scalability and annotation performance. In this research, in order to solve scalability and the image annotation time challenge, the prototype selection approach is used. The assumption of the prototype selection is based on single-label instances while, in image annotation, an instance has more than one label. It means that instances are multi-label. Hence, to employ prototype selection algorithms in image annotation, focusing on the concept of multi-label is a critical task. Thus, taking an appropriate measure in these methods to compute the rate of dissimilarity between label vectors has a great importance. The proposed approach in this paper is based on multi-labeling of prototype selection methods by selecting a modifying appropriate binary dissimilarity measure, in comparison two label vectors. The effectiveness of the proposed approach in reducing the number of training instances and selecting effective ones has been shown by experiments on large-scale NUS-WIDE family image sets. The experimental results showed the effectiveness of the proposed approach in reducing the number of instances and improving annotation performance.

On the suitability of Prototype Selection methods for kNN classification with distributed data

In the current Information Age, data production and processing demands are ever increasing. This has motivated the appearance of large-scale distributed information. This phenomenon also applies to Pattern Recognition so that classic and common algorithms, such as the k-Nearest Neighbour, are unable to be used. To improve the efficiency of this classifier, Prototype Selection (PS) strategies can be used. Nevertheless, current PS algorithms were not designed to deal with distributed data, and their performance is therefore unknown under these conditions. This work is devoted to carrying out an experimental study on a simulated framework in which PS strategies can be compared under classical conditions as well as those expected in distributed scenarios. Our results report a general behaviour that is degraded as conditions approach to more realistic scenarios. However, our experiments also show that some methods are able to achieve a fairly similar performance to that of the non-distributed scenario. Thus, although there is a clear need for developing specific PS methodologies and algorithms for tackling these situations, those that reported a higher robustness against such conditions may be good candidates from which to start.

Improving nearest neighbor classification using Ensembles of Evolutionary Generated Prototype Subsets

One of the most accurate types of prototype selection algorithms, preprocessing techniques that select a subset of instances from the data before applying nearest neighbor classification to it, are evolutionary approaches. These algorithms result in very high accuracy and reduction rates, but unfortunately come at a substantial computational cost. In this paper, we introduce a framework that allows to efficiently use the intermediary results of the prototype selection algorithms to further increase their accuracy performance. Instead of only using the fittest prototype subset generated by the evolutionary algorithm, we use multiple prototype subsets in an ensemble setting. Secondly, in order to classify a test instance, we only use prototype subsets that accurately classify training instances in the neighborhood of that test instance. In an experimental evaluation, we apply our new framework to four state-of-the-art prototype selection algorithms and show that, by using our framework, more accurate results are obtained after less evaluations of the prototype selection method. We also present a case study with a prototype generation algorithm, showing that our framework is easily extended to other preprocessing paradigms as well.

Prototype generation on structural data using dissimilarity space representation

Data reduction techniques play a key role in instance-based classification to lower the amount of data to be processed. Among the different existing approaches, prototype selection (PS) and prototype generation (PG) are the most representative ones. These two families differ in the way the reduced set is obtained from the initial one: While the former aims at selecting the most representative elements from the set, the latter creates new data out of it. Although PG is considered to delimit more efficiently decision boundaries, the operations required are not so well defined in scenarios involving structural data such as strings, trees, or graphs. This work studies the possibility of using dissimilarity space (DS) methods as an intermediate process for mapping the initial structural representation to a statistical one, thereby allowing the use of PG methods. A comparative experiment over string data is carried out in which our proposal is faced to PS methods on the original space. Results show that the proposed strategy is able to achieve significantly similar results to PS in the initial space, thus standing as a clear alternative to the classic approach, with some additional advantages derived from the DS representation.

Correcting the hub occurrence prediction bias in many dimensions

Data reduction is a common pre-processing step for k-nearest neighbor classification (kNN). The existing prototype selection methods implement different criteria for selecting relevant points to use in classification, which constitutes a selection bias. This study examines the nature of the instance selection bias in intrinsically high-dimensional data. In high-dimensional feature spaces, hubs are known to emerge as centers of influence in kNN classification. These points dominate most kNN sets and are often detrimental to classification performance. Our experiments reveal that different instance selection strategies bias the predictions of the behavior of hub-points in high-dimensional data in different ways. We propose to introduce an intermediate un-biasing step when training the neighbor occurrence models and we demonstrate promising improvements in various hubness-aware classification methods, on a wide selection of high-dimensional synthetic and real-world datasets.

최근접 이웃 규칙 기반 프로토타입 선택과 편의-분산을 이용한 성능 평가

최근접 이웃 규칙 기반 프로토타입 선택과 편의-분산을 이용한 성능 평가

SubMIL: Discriminative subspaces for multi-instance learning

As an important learning scheme for Multi-Instance Learning (MIL), the Instance Prototype (IP) selection-based MIL algorithms transform bags into a new instance feature space and achieve impressed classification performance. However, the number of IPs in the existing algorithms linearly increases with the scale of the training data. The performance and efficiencies of these algorithms are easily limited by the high dimension and noise when facing a large scale of training data. This paper proposes a discriminative subspaces-based instance prototype selection method that is suitable for reducing the computation complexity for large scale training data. In the proposed algorithm, we introduce the low-rank matrix recovery technique to find two discriminative and clean subspaces with less noise; then present a ℓ2,1 norm-based self-expressive sparse coding model to select the most representative instances in each subspace. Experimental results on several data sets show that our algorithm achieves superior and stable performance but with lower dimension compared with other IP selection strategies.

A Prototype Selection Method for Synthetic Object Recognition Based on Structural Graph Model

At present, many popular methods for object recognition are based on regional visual feature vectors which ignore the global structure of the object, leading to the “semantic gap” between image content and its tag. Graph-based structural representation of an object can record regional visual features and global relationships between regions that compose the object. However, the computation complexity of graph comparing limits the application of graph model in object recognition. One method for improving the recognizing speed is to reduce the prototypes in concept space of specific object category. We presented a prototype selection method for sythetic object recognition based on structural graph model using greedy clustering and representative sample selection. Experiments are conducted on a 2-D CAD synthetic object database. And the results show that the prototype selection method reduces the time cost of object recognition greatly with an acceptable loss in accuracy.

A SVM-based quantitative fMRI method for resting-state functional network detection

Resting-state functional magnetic resonance imaging (fMRI) aims to measure baseline neuronal connectivity independent of specific functional tasks and to capture changes in the connectivity due to neurological diseases. Most existing network detection methods rely on a fixed threshold to identify functionally connected voxels under the resting state. Due to fMRI non-stationarity, the threshold cannot adapt to variation of data characteristics across sessions and subjects, and generates unreliable mapping results. In this study, a new method is presented for resting-state fMRI data analysis. Specifically, the resting-state network mapping is formulated as an outlier detection process that is implemented using one-class support vector machine (SVM). The results are refined by using a spatial-feature domain prototype selection method and two-class SVM reclassification. The final decision on each voxel is made by comparing its probabilities of functionally connected and unconnected instead of a threshold. Multiple features for resting-state analysis were extracted and examined using an SVM-based feature selection method, and the most representative features were identified. The proposed method was evaluated using synthetic and experimental fMRI data. A comparison study was also performed with independent component analysis (ICA) and correlation analysis. The experimental results show that the proposed method can provide comparable or better network detection performance than ICA and correlation analysis. The method is potentially applicable to various resting-state quantitative fMRI studies.

Combining classifiers using nearest decision prototypes

We present a new classifier fusion method to combine soft-level classifiers with a new approach, which can be considered as a generalized decision templates method. Previous combining methods based on decision templates employ a single prototype for each class, but this global point of view mostly fails to properly represent the decision space. This drawback extremely affects the classification rate in such cases: insufficient number of training samples, island-shaped decision space distribution, and classes with highly overlapped decision spaces. To better represent the decision space, we utilize a prototype selection method to obtain a set of local decision prototypes for each class. Afterward, to determine the class of a test pattern, its decision profile is computed and then compared to all decision prototypes. In other words, for each class, the larger the numbers of decision prototypes near to the decision profile of a given pattern, the higher the chance for that class. The efficiency of our proposed method is evaluated over some well-known classification datasets suggesting superiority of our method in comparison with other proposed techniques.

FRPS: A Fuzzy Rough Prototype Selection method

The k Nearest Neighbour (k NN) method is a widely used classification method that has proven to be very effective. The accuracy of k NN can be improved by means of Prototype Selection (PS), that is, we provide k NN with a reduced but reinforced dataset to pick its neighbours from. We use fuzzy rough set theory to express the quality of the instances, and use a wrapper approach to determine which instances to prune. We call this method Fuzzy Rough Prototype Selection (FRPS) and evaluate its effectiveness on a variety of datasets. A comparison of FRPS with state-of-the-art PS methods confirms that our method performs very well with respect to accuracy.