Instance Selection Techniques Research Articles

A Comparative Survey of Instance Selection Methods applied to Non-Neural and Transformer-Based Text Classification

Progress in natural language processing has been dictated by the rule of more : more data, more computing power, more complexity, best exemplified by deep learning Transformers. However, training (or fine-tuning) large dense models for specific applications usually requires significant amounts of computing resources. One way to ameliorate this problem is through data engineering instead of the algorithmic or hardware perspectives. Our focus here is an under-investigated data engineering technique, with enormous potential in the current scenario – Instance Selection (IS) (a.k.a. Selective Sampling, Prototype Selection). The IS goal is to reduce the training set size by removing noisy or redundant instances while maintaining or improving the effectiveness (accuracy) of the trained models and reducing the training process cost. We survey classical and recent state-of-the-art IS techniques and provide a scientifically sound comparison of IS methods applied to an essential natural language processing task—Automatic Text Classification (ATC). IS methods have been normally applied to small tabular datasets and have not been systematically compared in ATC. We consider several neural and non-neural state-of-the-art ATC solutions and many datasets. We answer several research questions based on tradeoffs induced by a tripod (training set reduction, effectiveness, and efficiency). Our answers reveal an enormous unfulfilled potential for IS solutions. Specially, we show that in 12 out of 19 datasets, specific IS methods—namely, Condensed Nearest Neighbor, Local Set-based Smoother, and Local Set Border Selector—can reduce the size of the training set without effectiveness losses. Furthermore, in the case of fine-tuning the Transformer methods, the IS methods reduce the amount of data needed, without losing effectiveness and with considerable training-time gains.

An intuitionistic fuzzy bireduct model and its application to cancer treatment

Due to technological advancement, data size has seen a significant increase both in terms of features and instances. An efficient way to handle large sized datasets is to apply data reduction technique to ease the computational intractability of learning algorithms. Intuitionistic fuzzy rough and fuzzy rough sets based approaches provide a handful of solution, but the applicability is mostly restricted to reduction in terms of feature size. However, reduction in terms of instance size has not been handled by any of the existing intuionistic fuzzy set approaches. The present paper introduces the notion of bireducts in intuitionistic fuzzy framework that can be used for simultaneous reduction of instances and features. A robust lower approximation formulation is employed along with laying the foundation for the variants of instance selection technique. The proposed model is therefore robust to noise and can very effectively handle uncertainities due to judgement as well as identification. Further, an efficient instance selection technique in bireduct formulation enhances the performance. The experimental evaluation on benchmark datasets demonstrates the applicability and robustness of the proposed bireducts. It significantly reduces data size both in terms of instances and features whilst maintaining high performances. Further, the model is applied in the challenging domain of cancer treatment by enhancing the prediction performance of anti-angiogenic peptides. A comparative analysis demonstrates the superiority of the proposed methodology.

Weighted Ensemble with one-class Classification and Over-sampling and Instance selection (WECOI): An approach for learning from imbalanced data streams

Learning from imbalanced data streams is one of the challenges associated with classification algorithms and learning classifiers. The goal of this paper is to propose and validate a new approach for learning from data streams, with reference to the problem of class-imbalanced data. A hybrid approach for changing the class distribution towards a more balanced dataset using over-sampling and instance selection techniques is discussed. The proposed approach is based on the integration of a weighted ensemble classification and a technique to deal with the problem of class imbalance, and is called Weighted Ensemble with one-class Classification and Over-sampling and Instance selection (WECOI). Our approach assumes that classifiers are induced from incoming blocks of instances, called data chunks. These data chunks consist of incoming instances from different classes, and a balance between them is reached through our hybrid approach. These data chunks are then used to induce classifier ensembles. The proposed approach is validated experimentally using several selected benchmark datasets, and the results of computational experiments are presented and discussed. The results show that the proposed approach for eliminating class imbalance in data streams can help increase the performance of online learning algorithms. This is an extended version of a paper that was presented at the International Conference on Computational Science in 2021 (ICCS-2021) (Czarnowski, 2021). This version has been enriched with a deeper analysis of the results obtained from a modified variant of the originally proposed approach.

A Hybrid Machine Learning Model for Credit Approval

ABSTRACT Incorrect decision-making in financial institutions is very likely to cause financial crises. In recent years, many studies have demonstrated that artificial intelligence techniques can be used as alternative methods for credit scoring. Previous studies showed that prediction models built using hybrid approaches perform better than single approaches. In addition, feature selection or instance selection techniques should be incorporated into building prediction models to improve the prediction performance. In this study, we integrate feature selection, instance selection, and decision tree techniques to propose a new approach to predicting credit approval. Experimental results obtained using the survey data show that our proposed approach is superior to the other five traditional machine learning approaches in the measures. In addition, our approach has a lower cost effect than the traditional five methods. That is, the proposed approach generates fewer costs, such as money loss, than the traditional five approaches.

Generalization ability of Extreme Learning Machine using different Sample Selection Methods

The recent explosion of data has triggered the need of data reduction for completing the effective data mining task in the process of knowledge discovery in databases (KDD). The process of instance selection (IS) plays a significant role for data reduction by eliminating the redundant, noisy, unreliable and irrelevant instances, which, in-turn reduces the computational resources, and helps to increase the capabilities and generalization abilities of the learning models. . This manuscript expounds the concept and functionalities of seven different instance selection techniques (i.e., ENN, AllKNN, MENN, ENNTh, Mul- tiEdit, NCNEdit, and RNG), and also evaluates their effectiveness by using single layer feed-forward neural network (SLFN), which is trained with extreme learning machine (ELM). Unlike traditional neural network, ELM randomly chooses the weights and biases of hidden layer nodes and analytically determines the weights of output layer node. The generalization ability of ELM is analyzed by using both original and reduced datasets. Experiment results depict that ELM provides better generalization with these IS methods.

Combining feature selection, instance selection, and ensemble classification techniques for improved financial distress prediction

Bankruptcy prediction and credit scoring are major problems in financial distress prediction. Studies have shown that prediction models can be made more effective by performing data preprocessing procedures. Moreover, classifier ensembles are likely to outperform single classifiers. Although feature selection, instance selection, and classifier ensembles are known to affect final prediction results, their combined effects on bankruptcy prediction and credit scoring problems have not been fully explored. This study compares the performance of three feature selection algorithms, three instance selection algorithms, four classification algorithms, and two ensemble learning techniques. The results obtained using five bankruptcy prediction and five credit scoring datasets indicate that by carefully considering the combination of these three factors, better prediction models can be developed than by considering only one related factor.

Improvement the performance of the classification models of Cyclooxygenase-2 inhibitors using undersampling methods based on the rivality and reliability indexes

Undersampling, prototype or instance selection techniques are oriented to remove redundant and noisy molecules from the datasets in order to reduce the computational cost and the necessity of memory in the construction of QSAR models, maintaining the performance of the models. In this paper, we describe and apply an undersampling technique based on the rivality and reliability indexes to the building of classification models for two Cyclooxinase-2 inhibitors datasets. In a preprocessing stage, the datasets are analyzed and curated and classification models are built using Support Vector Machine, Random Forest and Rivality Index Neighborhood algorithms. The results obtained clearly improve the ones described in the literature for these datasets for the training models and external validations carried out. Values of Matthews Correlation Coefficient higher than 0.9 for the training models and external validations proved the high robustness of the models generated using the undersampling technique capable of generating reductions of the datasets greater than 80%.

Grey-based multiple instance learning with multiple bag-representative

Multiple instance learning is a modification in supervised learning that handles the classification of collection instances, which called bags. Each bag contains a number of instances whose features are extracted. In multiple instance learning, the standard assumption is that a positive bag contains at least one positive instance, whereas a negative bag is only comprised of negative instances. The complexity of multiple instance learning relies heavily on the number of instances in the training datasets. Since we are usually confronted with a large instance space, it is important to design efficient instance selection techniques to speed up the training process, without compromising the performance. Firstly, a multiple instance learning model of support vector machine based on grey relational analysis is proposed in this paper. The data size can be reduced, and the importance of instances in the bag can be preliminarily judged. Secondly, this paper introduces an algorithm with the bag-representative selector that trains the support vector machine based on bag-level information. Finally, this paper shows how to generalize the algorithm for binary multiple instance learning to multiple class tasks. The experimental study evaluates and compares the performance of our method against 8 state-of-the-art multiple instance methods over 10 datasets, and then demonstrates that the proposed approach is competitive with the state-of-art multiple instance learning methods.

Joint feature and instance selection using manifold data criteria: application to image classification

In many pattern recognition applications feature selection and instance selection can be used as two data preprocessing methods that aim at reducing the computational cost of the learning process. Moreover, in some cases, feature subset selection can improve the classification performance. Feature selection and instance selection can be interesting since the choice of features and instances greatly influence the performance of the learnt models as well as their training costs. In the past, unifying both problems was carried out by solving a global optimization problem using meta-heuristics. This paradigm not only does not exploit the manifold structure of data but can be computationally expensive. To the best of our knowledge, the joint use of sparse modeling representative and feature subset relevance have not been exploited by the joint feature and selection methods. In this paper, we target the joint feature and instance selection by adopting feature subset relevance and sparse modeling representative selection. More precisely, we propose three schemes for the joint feature and instance selection. The first is a wrapper technique while the two remaining ones are filter approaches. In the filter approaches, the search process adopts a genetic algorithm in which the evaluation is mainly given by a score that quantify the goodness of the features and instances. An efficient instance selection technique is used and integrated in the search process in order to adapt the instances to the candidate feature subset. We evaluate the performance of the proposed schemes using image classification where classifiers are the nearest neighbor classifier and support vector machine classifier. The study is conducted on five public image datasets. These experiments show the superiority of the proposed schemes over various baselines. The results confirm that the filter approaches leads to promising improvement on classification accuracy when both feature selection and instance selection are adopted.

Integrating Feature and Instance Selection Techniques in Opinion Mining

Opinion mining focuses on extracting polarity information from texts. For textual term representation, different feature selection methods, e.g. term frequency (TF) or term frequency–inverse document frequency (TF–IDF), can yield diverse numbers of text features. In text classification, however, a selected training set may contain noisy documents (or outliers), which can degrade the classification performance. To solve this problem, instance selection can be adopted to filter out unrepresentative training documents. Therefore, this article investigates the opinion mining performance associated with feature and instance selection steps simultaneously. Two combination processes based on performing feature selection and instance selection in different orders, were compared. Specifically, two feature selection methods, namely TF and TF–IDF, and two instance selection methods, namely DROP3 and IB3, were employed for comparison. The experimental results by using three Twitter datasets to develop sentiment classifiers showed that TF–IDF followed by DROP3 performs the best.

Prototype Selection Method Based on the Rivality and Reliability Indexes for the Improvement of the Classification Models and External Predictions.

Prototype or instance selection techniques is an important field of research in knowledge discovery, data mining, and machine learning. In QSAR, the use of prototype selection techniques in the preprocessing stage of the construction of the QSAR models favors the data set curation, improving the interpretability and accuracy of the models as well as the performance of the algorithms. In this paper, we propose an efficient method for prototype selection to be used in the preprocessing stage of the construction of QSAR classification models. The proposed method is able to generate very high reduction rates in the cardinality of the training set while maintaining or even increasing the accuracy of the classification models. The validation of the method has been carried out by means of the prediction of external molecules, demonstrating that the prediction of new molecules is also maintained or even improved. The method has been tested using 40 benchmark data sets of different sizes and balancing ratios; the results of the tests have demonstrated the wide applicability domain of the proposed method.

Ranking-based instance selection for pattern classification

In instance-based learning algorithms, the need to store a large number of examples as the training set results in several drawbacks related to large memory requirements, oversensitivity to noise, and slow execution speed. Instance selection techniques can improve the performance of these algorithms by selecting the best instances from the original data set, removing, for example, redundant information and noisy points. The relationship between an instance and the other patterns in the training set plays an important role and can impact its misclassification by learning algorithms. Such a relationship can be represented as a value that measures how difficult such instance is regarding classification purposes. Based on that, we introduce a novel instance selection algorithm called Ranking-based Instance Selection (RIS) that attributes a score per instance that depends on its relationship with all other instances in the training set. In this sense, instances with higher scores form safe regions (neighborhood of samples with relatively homogeneous class labels) in the feature space, and instances with lower scores form an indecision region (borderline samples of different classes). This information is further used in a selection process to remove instances from both safe and indecision regions that are considered irrelevant to represent their clusters in the feature space. In contrast to previous algorithms, the proposal combines a raking procedure with a selection process aiming to find a promising tradeoff between accuracy and reduction rate. Experiments are conducted on twenty-four real-world classification problems and show the effectiveness of the RIS algorithm when compared against other instance selection algorithms in the literature.

Ant colony optimization edge selection for support vector machine speed optimization

Support vector machine (SVM) is a widely used and reliable machine learning algorithm. It has been successfully applied to many real-world problems, with remarkable results. However, it has also been observed that SVM computational complexity increases with the increase in data size. Although many SVM speed optimization techniques have been proposed in the literature, there is still need for further improvement on the performance speed and accuracy of this algorithm. In this paper, a boundary detection algorithm for SVM speed optimization called ant colony optimization instance selection algorithm (ACOISA) is proposed. ACOISA is inspired by edge selection in ant colony optimization (ACO) algorithm, and it performs two primary functions: boundary detection and boundary instance selection. In the algorithm, ACO is used for boundary detection and k-nearest neighbor algorithm is used for boundary instance selection. Different sets of experiments are carried out to validate the efficiency of the proposed technique. All the experiments were performed on 35 datasets containing three well-known e-fraud types (credit card fraud, email spam and phishing email) and 31 other datasets available at UCI data repository. The experimental results showed that the proposed technique efficiently improved SVM training speed in 100% of the datasets used for evaluation, without significantly affecting SVM classification quality. Furthermore, the Freidman’s and Holm’s post hoc tests were conducted to statistically validate the credibility of the results. The statistical test results revealed that the proposed technique is significantly faster, compared to the standard SVM and some existing instance selection techniques, in all cases.

PAS3-HSID: a Dynamic Bio-Inspired Approach for Real-Time Hot Spot Identification in Data Streams

Hot spot identification is a very relevant problem in a wide variety of areas such as health care, energy or transportation. A hot spot is defined as a region of high likelihood of occurrence of a particular event. To identify hot spots, location data for those events is required, which is typically collected by telematics devices. These sensors are constantly gathering information, generating very large volumes of data. Current state-of-the-art solutions are capable of identifying hot spots from big static batches of data by means of variations of clustering or instance selection techniques that pre-process the original input data, providing the most relevant locations. However, these approaches neglect to address changes in hot spots over time. This paper presents a dynamic bio-inspired approach to detect hot spots in big data streams. This computational intelligence method is designed and applied to the transportation sector as a case study to identify incidents in the roads caused by heavy goods vehicles. We adapt an immune-based algorithm to account for the temporary aspect of hot spots inspired by the idea of pheromones, which is then subsequently implemented using Apache Spark Streaming. Experimental results on real datasets with up to 4.5 million data points—provided by a telematics company—show that the algorithm is capable of quickly processing large streaming batches of data, as well as successfully adapting over time to detect hot spots. The outcome of this method is twofold, both reducing data storage requirements and demonstrating resilience to sudden changes in the input data (concept drift).

Nature Inspired Instance Selection Techniques for Support Vector Machine Speed Optimization

Due to the fast-growing rate of information sources, many organizations and individuals are overwhelmed with vast amounts of data. The rate of data growth is very alarming, and it is already going beyond the Exabyte limit. Hence, there is an obvious need for fast and accurate big data classification tools. Machine Learning (ML) based solutions are very useful and reliable data classification tools, however, they cannot effectively handle large-scale datasets. This paper therefore proposes two intelligent instance selection techniques for optimizing the training and classification speed of ML algorithms, with a specific focus on Support Vector Machine (SVM). Furthermore, this paper considers two different approaches to instance selection namely: filter-based and wrapper-based. Different sets of experiments are performed on 20 small-scale datasets and 10 large- or medium-scale datasets. The results show that the proposed techniques improved SVM training speed in 100% (30 out of 30) of the datasets used for evaluation and simultaneously improved SVM predictive accuracy in some cases. Furthermore, statistical analysis test is carried out and the results reveal that the training speed of the proposed techniques is statistically significantly faster than the training speed of standard SVM and some other existing instance selection techniques. In real life application, such as video surveillance and intrusion detection systems, that require a classifier to be trained quickly for speedy classification of new target concepts, the filter-based techniques provide the best solutions; while the wrapper-based techniques are better suited for applications such as email filters, that are very sensitive to slight changes in predictive accuracy.

Combining instance selection and self-training to improve data stream quantification

In the last years, learning from data streams has attracted the attention of researchers and practitioners due to its large number of applications. These applications have motivated the research community to propose a significant amount of methods to solve problems in diverse tasks, more prominently in classification, clustering, and anomaly detection. However, a relevant task known as quantification has remained mostly unexplored. The quantification goal is to provide an estimate of the class prevalence in an unlabeled set. Recently, we proposed the SQSI algorithm to quantify data streams with concept drifts. SQSI uses a statistical test to identify concept drifts and retrain the classifiers. However, the retraining involves requiring the labels for all newly arrived instances. In this paper, we extend SQSI algorithm by exploring instance selection techniques allied to semi-supervised learning. The idea is to request the classes of a smaller subset of recent examples. Our experiments demonstrate that SQSI’s extension significantly reduces the dependency on actual labels while maintaining or improving the quantification accuracy.

Local sets for multi-label instance selection

The multi-label classification problem is an extension of traditional (single-label) classification, in which the output is a vector of values rather than a single categorical value. The multi-label problem is therefore a very different and much more challenging one than the single-label problem. Recently, multi-label classification has attracted interest, because of its real-life applications, such as image recognition, bio-informatics, and text categorization, among others. Unfortunately, there are few instance selection techniques capable of processing the data used for these applications. These techniques are also very useful for cleaning and reducing the size of data sets.In single-label problems, the local set of an instance x comprises all instances in the largest hypersphere centered on x, so that they are all of the same class. This concept has been successfully integrated in the design of Iterative Case Filtering, one of the most influential instance selection methods in single-label learning. Unfortunately, the concept that was originally defined for single-label learning cannot be directly applied to multi-label data, as each instance has more than one label.An adaptation of the local set concept to multi-label data is proposed in this paper and its effectiveness is verified in the design of two new algorithms that yielded competitive results. One of the adaptations cleans the data sets, to improve their predictive capabilities, while the other aims to reduce data set sizes. Both are tested and compared against the state-of-the-art instance selection methods available for multi-label learning.

On the Performance of Cuckoo Search and Bat Algorithms Based Instance Selection Techniques for SVM Speed Optimization with Application to e-Fraud Detection

On the Performance of Cuckoo Search and Bat Algorithms Based Instance Selection Techniques for SVM Speed Optimization with Application to e-Fraud Detection

DEMIAL: an active learning framework for multiple instance image classification using dictionary ensembles

In many machine-learning applications, each data point can be represented as a set of instances that create multiple instance learning (MIL) problems. Due to the structure of images, different regions can be interpreted as instances. Thus, multiple instances can be obtained for each image, which makes image categorization a MIL problem. With abundant unlabeled image data, this MIL problem can be solved using active learning algorithms. Active learning is a framework that utilizes unlabeled data in which labeling samples is a labor-intensive and expensive task. Although many effective MIL active learning methods have been developed, most of the existing algorithms do not take into account classifier and feature representation. In this work, we develop DEMIAL (dictionary ensembles multiple instance active learning), a multiple instance active learning method that utilizes sparse feature representation and classifier ensemble techniques. In the proposed active learning framework, we employ dictionary learning and compare uncertainty- and entropy-based instance selection techniques. Experimental results show that classifier ensembles benefit from active learning and the DEMIAL algorithm outperforms the kernel-based multiple instance active learning framework.

Instance Selection Techniques in Reduction of Data Streams Derived from Medical Devices

Instance Selection Techniques in Reduction of Data Streams Derived from Medical Devices