Prototype Reduction Research Articles

PRLDPC: A Heuristics Prototype Reduction Method Based on Supervised Local Density Clustering for Instance-Based Classifiers

The prototype reduction (PR) methods, as an important data pre-processing task, can improve instance-based classifiers by removing noise and/or redundant samples. Recently, a series of PR methods with different heuristic strategies have been developed. Among them, clustering-based PR methods have shown competitive performance. Yet, they still suffer from the following issues: (a) most methods heavily rely on parameters; (b) most fail to remove suspicious noisy samples from the training set; (c) almost all fail to handle manifold data with nonspherical distributions effectively; (d) some have a relatively high time complexity. To advance the state of the art of clustering-based PR methods by overcoming the above issues, a novel heuristics PR method based on supervised local density peaks clustering (PRLDPC) is proposed. The main ideas of PRLDPC are concluded as follows: (a) a supervised local density peaks clustering (SLDPC) is first proposed to divide the training set into homogeneous and heterogeneous sub-clusters; (b) SLDPC-based edition method is second proposed to identify and remove noisy samples from heterogeneous sub-clusters; (c) an SLDPC-based condensing method is third proposed to obtain reduced samples from homogeneous sub-clusters and pruned heterogeneous sub-clusters. Intensive experiments have proven that (a) PRLDPC can outperform six state-of-the-art PR methods on extensive UCI and Kaggle data sets in weighing the reduction rate and classification accuracy of three instance-based classifiers; (b) PRLDPC is relatively fast and has a relatively low time complexity [Formula: see text].

Instance-based learning with prototype reduction for real-time proportional myocontrol: a randomized user study demonstrating accuracy-preserving data reduction for prosthetic embedded systems

This work presents the design, implementation and validation of learning techniques based on the kNN scheme for gesture detection in prosthetic control. To cope with high computational demands in instance-based prediction, methods of dataset reduction are evaluated considering real-time determinism to allow for the reliable integration into battery-powered portable devices. The influence of parameterization and varying proportionality schemes is analyzed, utilizing an eight-channel-sEMG armband. Besides offline cross-validation accuracy, success rates in real-time pilot experiments (online target achievement tests) are determined. Based on the assessment of specific dataset reduction techniques’ adequacy for embedded control applications regarding accuracy and timing behaviour, decision surface mapping (DSM) proves itself promising when applying kNN on the reduced set. A randomized, double-blind user study was conducted to evaluate the respective methods (kNN and kNN with DSM-reduction) against ridge regression (RR) and RR with random Fourier features (RR-RFF). The kNN-based methods performed significantly better (p < 0.0005) than the regression techniques. Between DSM-kNN and kNN, there was no statistically significant difference (significance level 0.05). This is remarkable in consideration of only one sample per class in the reduced set, thus yielding a reduction rate of over 99% while preserving success rate. The same behaviour could be confirmed in an extended user study. With k=1, which turned out to be an excellent choice, the runtime complexity of both kNN (in every prediction step) as well as DSM-kNN (in the training phase) becomes linear concerning the number of original samples, favouring dependable wearable prosthesis applications.Graphical abstract

Deep Metric Learning for K Nearest Neighbor Classication

KNN has gained popularity in machine learning due to its simplicity and good performance. However, kNN faces two problems with classification tasks. The first is that an appropriate distance measurement is required to compute distances between test sample and training samples. The other is the highly computational complexity due to the requirement of searching the nearest neighbors in the whole training data. In order to mitigate these two problems, we propose a novel method named KCNN to enhance the performance of kNN. KCNN uses convolutional neural networks to learn a suitable distance metric as well as prototype reduction to learn a reduced set of prototypes which can represent the original set. It has several superiorities compared with related methods. The combination of CNN and kNN empowers it to extract discriminative hierarchical features with which kNN can easily classify. KCNN learns spatial information on an image instead of considering it as a vector to learn distance metric. Moreover, KCNN simultaneously learns a reduced set of prototypes, which help improve classification efficiency and avoid noisy samples of the massive training set. The proposed method has a better robustness and convergence than CNN, especially when projecting input data into a low-dimension space.

ConvProtoNet: Deep Prototype Induction towards Better Class Representation for Few-Shot Malware Classification

Traditional malware classification relies on known malware types and significantly large datasets labeled manually which limits its ability to recognize new malware classes. For unknown malware types or new variants of existing malware containing only a few samples each class, common classification methods often fail to work well due to severe overfitting. In this paper, we propose a new neural network structure called ConvProtoNet which employs few-shot learning to address the problem of scarce malware samples while prevent from overfitting. We design a convolutional induction module to replace the insufficient prototype reduction in most few-shot models and generates more appropriate class-level malware prototypes for classification. We also adopt meta-learning scheme to make classifier robust enough to adapt unseen malware classes without fine-tuning. Even in extreme conditions where only 5 samples in each class are provided, ConvProtoNet still achieves more than 70% accuracy on average and outperforms other traditional malware classification methods or existed few-shot models in experiments conducted on several datasets. Extra experiments across datasets illustrate that ConvProtoNet learns general knowledge of malware which is dataset-invariant and careful model analysis proves effectiveness of ConvProtoNet in few-shot malware classification.

Unsupervised prototype reduction for data exploration and an application to air traffic management initiatives

We discuss a new approach to unsupervised learning and data exploration that involves summarizing a large data set using a small set of “representative” elements. These representatives may be presented to a user in order to provide intuition regarding the distribution of observations. Alternatively, these representatives can be used as cases for more detailed analysis. We call the problem of selecting the representatives the unsupervised prototype reduction problem. We discuss the KC-UPR method for this problem and compare it to other existing methods that may be applied to this problem. We propose a new type of distance measure that allows for more interpretable presentation of results from the KC-UPR method. We demonstrate how solutions from the unsupervised prototype reduction problem may be used to provide decision support for the planning of air traffic management initiatives, and we produce computational results that compare the effectiveness of several methods in this application. We also provide an example of how the KC-UPR method can be used for data exploration, using data from air traffic management initiatives at Newark Liberty International Airport.

MODEL PENGENTASAN KEMISKINAN TERINTEGRASI DI KOTA YOGYAKARTA

This research aim to find model reduction poorness of integrated in Yogyakarta testing having taken steps in three subdistrict that is Kricak, Tegalpanggung and Sorosutan . Research method by using research model and expansion with antecedent study stages; steps, expansion prototype , field test and semination product result of expansion. Antecedent study by doing in-depth interview and observation to poor family , and public figures and studies documents ( former research, institute report) which with reference to reduction poorness. Result of firststep study in the form of information of aspiration of poor family, the Government plan and Non Government Organization and research conclusion before all, utilized isn't it to compile expansion of model prototype reduction poorness. Step of hereinafter test prototype model reduction poorness of three sub-district Kricak, Tegalpanggung and Sorosutan to obtain empiric evidence about elegibility of execution process from model limitedly, either its(the subject and also aspects). Result of his its would in semination to some other sub-districts.Its the target is find enableness model of public in the form of expansion of model reduction poorness integrated, mean from various aspects ( economics, social, education, health, culture) and various element considered and its the execution is really can lessen number of poor families Kata Kunci: kemiskinan, Yogyakarta, integrasi

CHI-PG: A fast prototype generation algorithm for Big Data classification problems

Abstract The growing amount of available data has become a serious challenge to data mining and machine learning techniques. Well-known classification methods that have been widely applied so far are no longer feasible in Big Data environments. For this reason, prototype reduction techniques (both selection and generation) come up as a candidate solution to build a reduced version of the dataset that speeds up the execution of algorithms such as k-Nearest Neighbors and overcome their memory constraints. However, these solutions generally have a quadratic O(N2) time complexity and share similar limitations to those encountered in data mining and machine learning algorithms in terms of time and memory requirements. In order to overcome these limitations, we introduce a new distributed MapReduce prototype generation method called CHI-PG that provides a linear O(N) time complexity and ensures constant accuracy regardless of the degree of parallelism. This approach builds prototypes by applying a simple scheme based on the rule generation process of the Chi et al. Fuzzy Rule-Based Classification System and takes advantage of the suitability of this classifier for the MapReduce paradigm. The empirical study shows that our new approach significantly improves the execution time of a state-of-the-art distributed prototype reduction algorithm (MRPR) without decreasing (and even improving) classification accuracy and reduction rates. Moreover, CHI-PG has been shown to be a candidate solution to the time and memory constraints of k-Nearest Neighbors when tackling large-scale datasets.

Unsupervised Prototype Reduction for Data Exploration and An Application to Air Traffic Management Initiatives

We discuss a new approach to unsupervised learning and data exploration that involves summarizing a large data set using a small set of ``representative'' elements. These representatives may be presented to a user in order to provide intuition regarding the distribution of observations. Alternatively, these representatives can be used as cases for more detailed analysis. We call the problem of selecting the representatives the Unsupervised Prototype Reduction problem. We discuss the KC-UPR method for this problem and compare it to other existing methods that may be applied to this problem. We propose a new type of distance measure that allows for more interpretable presentation of results from the KC-UPR method. We demonstrate how solutions from the Unsupervised Prototype Reduction problem may be used to provide decision support for the planning of air traffic management initiatives, and we produce computational results that compare the effectiveness of several methods in this application. We also provide an example of how the KC-UPR method can be used for data exploration, using data from air traffic management initiatives at Newark Liberty International Airport.

EPRENNID: An evolutionary prototype reduction based ensemble for nearest neighbor classification of imbalanced data

Classification problems with an imbalanced class distribution have received an increased amount of attention within the machine learning community over the last decade. They are encountered in a growing number of real-world situations and pose a challenge to standard machine learning techniques. We propose a new hybrid method specifically tailored to handle class imbalance, called EPRENNID. It performs an evolutionary prototype reduction focused on providing diverse solutions to prevent the method from overfitting the training set. It also allows us to explicitly reduce the underrepresented class, which the most common preprocessing solutions handling class imbalance usually protect. As part of the experimental study, we show that the proposed prototype reduction method outperforms state-of-the-art preprocessing techniques. The preprocessing step yields multiple prototype sets that are later used in an ensemble, performing a weighted voting scheme with the nearest neighbor classifier. EPRENNID is experimentally shown to significantly outperform previous proposals.

Selecting promising classes from generated data for an efficient multi-class nearest neighbor classification

The nearest neighbor rule is one of the most considered algorithms for supervised learning because of its simplicity and fair performance in most cases. However, this technique has a number of disadvantages, being the low computational efficiency the most prominent one. This paper presents a strategy to overcome this obstacle in multi-class classification tasks. This strategy proposes the use of Prototype Reduction algorithms that are capable of generating a new training set from the original one to try to gather the same information with fewer samples. Over this reduced set, it is estimated which classes are the closest ones to the input sample. These classes are referred to as promising classes. Eventually, classification is performed using the original training set using the nearest neighbor rule but restricted to the promising classes. Our experiments with several datasets and significance tests show that a similar classification accuracy can be obtained compared to using the original training set, with a significantly higher efficiency.

An experimental study on rank methods for prototype selection

Prototype selection is one of the most popular approaches for addressing the low efficiency issue typically found in the well-known k-Nearest Neighbour classification rule. These techniques select a representative subset from an original collection of prototypes with the premise of maintaining the same classification accuracy. Most recently, rank methods have been proposed as an alternative to develop new selection strategies. Following a certain heuristic, these methods sort the elements of the initial collection according to their relevance and then select the best possible subset by means of a parameter representing the amount of data to maintain. Due to the relative novelty of these methods, their performance and competitiveness against other strategies is still unclear. This work performs an exhaustive experimental study of such methods for prototype selection. A representative collection of both classic and sophisticated algorithms are compared to the aforementioned techniques in a number of datasets, including different levels of induced noise. Results report the remarkable competitiveness of these rank methods as well as their excellent trade-off between prototype reduction and achieved accuracy.

Pattern classification using a new border identification paradigm: The nearest border technique

There are many paradigms for pattern classification such as the optimal Bayesian, kernel-based methods, inter-class border identification schemes, nearest neighbor methods, nearest centroid methods, among others. As opposed to these, this paper pioneers a new paradigm, which we shall refer to as the nearest border (NB) paradigm. The philosophy for developing such a NB strategy is as follows: given the training data set for each class, we shall attempt to create borders for each individual class. However, unlike the traditional border identification (BI) methods, we do not undertake this by using inter-class criteria; rather, we attempt to obtain the border for a specific class in the d-dimensional hyper-space by invoking only the properties of the samples within that class. Once these borders have been obtained, we advocate that testing is accomplished by assigning the test sample to the class whose border it lies closest to. This claim appears counter-intuitive, because unlike the centroid or the median, these border samples are often “outliers” and are, really, the points that represent the class the least. Moreover, inter-class BI methods intuitively outperform within-class ones. However, we have formally proven this claim, and the theoretical results have been verified by rigorous experimental testing on artificial and real-life data sets. While the solution we propose is distantly related to the reported solutions involving prototype reduction schemes (PRSs) and BI algorithms, it is, most importantly, akin to the recently proposed “anti-Bayesian” methods of classification.

MRPR: A MapReduce solution for prototype reduction in big data classification

In the era of big data, analyzing and extracting knowledge from large-scale data sets is a very interesting and challenging task. The application of standard data mining tools in such data sets is not straightforward. Hence, a new class of scalable mining method that embraces the huge storage and processing capacity of cloud platforms is required. In this work, we propose a novel distributed partitioning methodology for prototype reduction techniques in nearest neighbor classification. These methods aim at representing original training data sets as a reduced number of instances. Their main purposes are to speed up the classification process and reduce the storage requirements and sensitivity to noise of the nearest neighbor rule. However, the standard prototype reduction methods cannot cope with very large data sets. To overcome this limitation, we develop a MapReduce-based framework to distribute the functioning of these algorithms through a cluster of computing elements, proposing several algorithmic strategies to integrate multiple partial solutions (reduced sets of prototypes) into a single one. The proposed model enables prototype reduction algorithms to be applied over big data classification problems without significant accuracy loss. We test the speeding up capabilities of our model with data sets up to 5.7 millions of instances. The results show that this model is a suitable tool to enhance the performance of the nearest neighbor classifier with big data.

A new fast reduction technique based on binary nearest neighbor tree

The K-nearest neighbor ( KNN ) rule is one of the most useful supervised classification methods, and is widely used in many pattern classification applications due to its simplicity. However, it faces prohibitive computational and storage requirements when dealing with large datasets. A reasonable way of alleviating this problem is to extract a small representative subset from the original dataset without reducing the classification accuracy. This means the most internal patterns are removed and the boundary patterns that can contribute to better classification accuracy are retained. To achieve this purpose, a new algorithm based on binary tree technique and some reduction operations is presented. The key issues of the proposed algorithm are how to build binary nearest neighbor search tree and design reduction strategies to keep the high classification accuracy patterns. In particular, firstly, we utilize several tree control rules and KNN rule to build a binary nearest neighbor tree of each random pattern. Secondly, according to the node locations in each binary nearest neighbor tree and the strategies of selection and replacement, different kinds of patterns as prototypes are obtained, which are close to class boundary regions or locate in the interior regions, and some internal patterns are generated. Finally, experimental results show that the proposed algorithm effectively reduces the number of prototypes while maintaining the same level of classification accuracy as the traditional KNN algorithm and other prototype algorithms. Moreover, it is a simple and fast hybrid algorithm for prototype reduction.

One-Class Classification based on searching for the problem features limits

Novelty detection is a problem with a large number of relevant applications. For some applications, the main interest is the prevention or detection of undesired states. In some cases, these undesired states are not known in advance; in others, such as machine monitoring, for instance, machine breakdown may be very rare and examples of this event may be unavailable. In such cases, the most widely accepted approach is to model the normal behavior of the system in order to subsequently detect unknown events. This is the basic concept of One-Class Classification (OCC).In some preliminary works we have proposed the Feature Boundaries Detector for One-Class Classification FBDOCC method, which operates by examining each problem feature at a time. In this paper we propose an extensive study of the behavior of the FBDOCC and introduce another version of the method, namely FBDOCC2. This work, also considers the use of the Particle Swarm Optimization (PSO) algorithm to find the best parameter configuration of the proposed method. Furthermore, this work also introduces a procedure to improve the training time performance without degrading the quality of the classification as well as some other contributions hereafter described.A number of experiments were carried out with synthetic and real datasets aiming at comparing both versions of the FBDOCC method with the most recent and effective OCC methods, namely: SVDD, One-class SVM, Least Squares One-class SVM, Kernel PCA, Gaussian Process Prior OCC, Condensed Nearest Neighbor Data Description and One-class Random Forests. The evaluation metrics considered in the experiments are: (i) the Area Under the ROC Curve (AUC); (ii) the Matthews Correlation Coefficient (MCC); (iii) the training time; and (iv) the prototype reduction rate. Regarding the metrics AUC and MCC, the first FBDOCC version has presented the best overall results among all the methods whereas the FBDOCC2 obtained results comparable to the best methods in some experiments where the standard FBDOCC yielded a poorer performance. FBDOCC was the faster method to train in comparison to the other methods in all but one dataset. In addition, FBDOCC was much faster than all methods based on support vector machines.

Prototype reduction based on Direct Weighted Pruning

Instance-based learning methods often suffer from problems related to high storage requirements, large computational costs for searching through the stored instances to find the ones most similar to the queries, and also sensitivity to noisy samples. In order to deal with these issues, various condensation algorithms have been proposed in the literature to reduce the set of prototypes that need to be stored. In this paper, we propose a new algorithm that uses a set of weights to directly control which prototypes have to be discarded or survive. Instead of relying on indirect heuristics, it explicitly optimizes a bi-objective index which incorporates the condensation rate and a measure of the classification inaccuracy as reflected by the nearest neighbor rule. The proposed algorithm, referred to as DWP (Direct Weighted Pruning), performs an efficient search using a simple genetic algorithm, which is however equipped with three novel acceleration mechanisms to notably speed up its convergence. Experiments over a large number of datasets and comparisons against many other successful condensation algorithms, show that DWP is very effective and achieves the highest classification accuracy along with competitive condensation rates.

A fast prototype reduction method based on template reduction and visualization-induced self-organizing map for nearest neighbor algorithm

The k nearest neighbor is a lazy learning algorithm that is inefficient in the classification phase because it needs to compare the query sample with all training samples. A template reduction method is recently proposed that uses only samples near the decision boundary for classification and removes those far from the decision boundary. However, when class distributions overlap, more border samples are retrained and it leads to inefficient performance in the classification phase. Because the number of reduced samples are limited, using an appropriate feature reduction method seems a logical choice to improve classification time. This paper proposes a new prototype reduction method for the k nearest neighbor algorithm, and it is based on template reduction and ViSOM. The potential property of ViSOM is displaying the topology of data on a two-dimensional feature map, it provides an intuitive way for users to observe and analyze data. An efficient classification framework is then presented, which combines the feature reduction method and the prototype selection algorithm. It needs a very small data size for classification while keeping recognition rate. In the experiments, both of synthetic and real datasets are used to evaluate the performance. Experimental results demonstrate that the proposed method obtains above 70 % speedup ratio and 90 % compression ratio while maintaining similar performance to kNN.

A Taxonomy and Experimental Study on Prototype Generation for Nearest Neighbor Classification

The nearest neighbor (NN) rule is one of the most successfully used techniques to resolve classification and pattern recognition tasks. Despite its high classification accuracy, this rule suffers from several shortcomings in time response, noise sensitivity, and high storage requirements. These weaknesses have been tackled by many different approaches, including a good and well-known solution that we can find in the literature, which consists of the reduction of the data used for the classification rule (training data). Prototype reduction techniques can be divided into two different approaches, which are known as prototype selection and prototype generation (PG) or abstraction. The former process consists of choosing a subset of the original training data, whereas PG builds new artificial prototypes to increase the accuracy of the NN classification. In this paper, we provide a survey of PG methods specifically designed for the NN rule. From a theoretical point of view, we propose a taxonomy based on the main characteristics presented in them. Furthermore, from an empirical point of view, we conduct a wide experimental study that involves small and large datasets to measure their performance in terms of accuracy and reduction capabilities. The results are contrasted through nonparametrical statistical tests. Several remarks are made to understand which PG models are appropriate for application to different datasets.

Cellular automata based nearest neighbour rule condensation

Most of the prototype reduction algorithms process the data in its entirety to yield a consistent subset, which is very useful in nearest neighbour classification. Their main disadvantage is the excessive computational cost when the prototype size is very large. In this paper, we present a cellular automata (CA)-based nearest neighbour rule condensation method to reduce useless points in a given training set. This method retains only the points on the boundary between different classes, and the amount of the reduced rules of the reference set can be revised by the granularity of the CA lattice. The main advantages of the proposed method are, on the one hand, that it is able to condense a given rule set within less time compared to other traditional algorithms. On the other hand, with the proposed algorithm, we can get a consistent subset of the given set in the divide-reduce-coalesce manner. Experiments show successful results when the size of the given dataset is large.

On using prototype reduction schemes to optimize locally linear reconstruction methods

This paper concerns the use of prototype reduction schemes (PRS) to optimize the computations involved in typical k-nearest neighbor (k-NN) rules. These rules have been successfully used for decades in statistical pattern recognition (PR) [1,15] applications and are particularly effective for density estimation, classification, and regression because of the known error bounds that they possess. For a given data point of unknown identity, the k-NN possesses the phenomenon that it combines the information about the samples from a priori target classes (values) of selected neighbors to predict the target class of the tested sample, or to estimate the density function value of the given queried sample. Recently, an implementation of the k-NN, named as the locally linear reconstruction (LLR) [2], has been proposed. The salient and brilliant feature of the latter is that by invoking a quadratic optimization process, it is capable of systematically setting model parameters, such as the number of neighbors (specified by the parameter, k) and the weights. However, the LLR takes more time than other conventional methods when it has to be applied to classification tasks. To overcome this problem, we propose a strategy of using a PRS to efficiently compute the optimization problem. In this paper, we demonstrate, first of all, that by completely discarding the points not included by the PRS, we can obtain a reduced set of sample points, using which, in turn, the quadratic optimization problem can be computed far more expediently. The values of the corresponding indices are comparable to those obtained with the original training set (i.e., the one which considers all the data points) even though the computations required to obtain the prototypes and the corresponding classification accuracies are noticeably less. The proposed method has been tested on artificial and real-life data sets, and the results obtained are very promising, and could have potential in PR applications.