K-nearest Neighbor Rule Research Articles

A fault isolation strategy for industrial processes using outlier-degree-based variable contributions

In industrial process monitoring, it is always a challenging and practical problem to analyze the causes of the system fault by isolating true fault variables from vast amounts of process data. However, the phenomenon of smearing effect occurs by using the traditional contribution analysis-based isolation methods since the defined isolation indices of different variables affect each other. In this paper, a new fault isolation method is proposed based on local outlier factor and improved k-nearest neighbor rule aiming to improve the isolation accuracy. Firstly, the nearest neighbors of each sample are obtained along the direction of a specific variable. Based on the nearest neighbors, the outlier-degree value of the variable is calculated and regarded as the contribution of the variable. Then, the contribution of the variable in all samples are obtained in the same way, among which the maximum one is selected as the isolation threshold value of this variable. During the online monitoring, the contribution of the variable in the newly collected sample is calculated in real time. Once the contribution is greater than the threshold, the variable is judged to be the dominant factor causing the system fault. Two cases on numerical example and Tennessee Eastman process are conducted to evaluate the effectiveness of the proposed method.

Early Warning Method of Structural Damage Using Localized Frequency Cointegration under Changing Environments

Modal frequencies are widely used to capture dynamics and reveal possible failures of bridge structures. However, the non-Gaussianity and nonlinearity of nonstationary modal frequencies associated with the actual structure under changing environmental conditions often restrict the application in structural health monitoring. Therefore, a new localized frequency cointegration without environmental measurements for the elimination of environmental interference and damage warning is proposed in this paper. The k-nearest neighbor rule is first employed to search for sufficient nearest neighbors for each modal feature based on similarity measurements over a wide range of training data. After that, the cointegration analysis associated with the Johansen procedure is performed to remove environmental trends and obtain multivariable cointegration residuals. Then, the defined early warning index (i.e., the weighted Mahalanobis distance) derived from the exponentially weighted moving average is adopted with respect to the identification of subtle damage. Eight groups of cointegration models between nonstationary modal variables are constructed and validated on the Z24 bridge case. The results demonstrate that the proposed approach can successfully discard spurious influences of environmental variations and identify the occurrence of real structural damage compared to traditional methods. Additionally, this approach is not constrained by the statistical distribution of observation samples, and the selection and combination of different modal orders are crucial in capturing the changes in frequency anomalies for the bridge condition assessment.

Consistency of the k-nearest neighbors rule for functional data

The problem of nonparametric classification by k-nearest neighbors rule in a general metric space will be considered. Consistency and strong consistency of the classifier will be established under mild conditions.

ML-k’sNN: Label Dependent k Values for Multi-Label k-Nearest Neighbor Rule

Multi-label classification as a data mining task has recently attracted increasing interest from researchers. Many current data mining applications address problems with instances that belong to more than one category. These problems require the development of new, efficient methods. Multi-label k-nearest neighbors rule, ML-kNN, is among the best-performing methods for multi-label problems. Current methods use a unique k value for all labels, as in the single-label method. However, the distributions of the labels are frequently very different. In such scenarios, a unique k value for the labels might be suboptimal. In this paper, we propose a novel approach in which each label is predicted with a different value of k. Obtaining the best k for each label is stated as an optimization problem. Three different algorithms are proposed for this task, depending on which multi-label metric is the target of our optimization process. In a large set of 40 real-world multi-label problems, our approach improves the results of two different tested ML-kNN implementations.

Local-based [formula omitted] values for multi-label [formula omitted]-nearest neighbors rule

Multi-label learning is a growing field in machine learning research. Many applications address instances that simultaneously belong to many categories, which cannot be disregarded if optimal results are desired. Among the many algorithms developed for multi-label learning, the multi-label k-nearest neighbor method is among the most successful. However, in a difficult classification task, such as multi-label learning, a challenge that arises in the k-nearest neighbor approach is the assignment of the appropriate value of k. Although a suitable value might be obtained using cross-validation, it is unlikely that the same value will be optimal for the whole space spanned by the training set. It is evident that different regions of the feature space would have different distributions of instances and labels that would require different values of k. The very complex boundaries among the many present labels make the necessity of local k values even more important than in the case with a single-label k-nearest neighbor. We present a simple yet powerful approach for setting a local value of k. We associate a potentially different k with every prototype and obtain the best value of that k by optimizing the criterion consisting of the local effect of the different k values in the neighborhood of the prototype. The proposed method has a fast training stage, as it only uses the neighborhood of each training instance to set the local k value. The complexity of the proposed method in terms of the testing time is similar to that of the standard multi-label k-nearest neighbor approach. Experiments performed on a set of 20 problems show that not only does our proposed method significantly outperform the standard multi-label k-nearest neighbor rule but also the locally adaptive multi-label k-nearest neighbor method can benefit from a local k value.

Consistency of the k-Nearest Neighbor Classifier for Spatially Dependent Data

The purpose of this paper is to investigate the k-nearest neighbor classification rule for spatially dependent data. Some spatial mixing conditions are considered, and under such spatial structures, the well known k-nearest neighbor rule is suggested to classify spatial data. We established consistency and strong consistency of the classifier under mild assumptions. Our main results extend the consistency result in the i.i.d. case to the spatial case.

Fault detection and diagnosis strategy based on k-nearest neighbors and fuzzy C-means clustering algorithm for industrial processes

Fault detection and diagnosis is crucial in recent industry sector to ensure safety and reliability, and improve the overall equipment efficiency. Moreover, fault detection and diagnosis based on k-nearest neighbor rule (FDD-kNN) has been effectively applied in industrial processes with characteristics such as multi-mode, non-linearity, and non-Gaussian distributed data. The main challenge associated with FDD-kNN is the on-line computational complexity and storage space that are needed for searching neighbors. To deal with these issues, this paper proposes a monitoring approach where the Fuzzy C-Means clustering technique is used to decrease the overall on-line computations and required storage by measuring the neighbors of the clusters’ centres rather than the raw data. After building the monitoring model off-line based on the data clusters’ centres, the faults are detected by comparing the average squared Euclidean distance between the on-line data sample and the clusters’ centres with a predefined threshold. Then, the detected faults can be diagnosed by calculating the contribution of each variable in the fault detection index. Furthermore, for easily analysing the fault diagnosis results, the relative contribution for each sample data vector is considered. A numerical example and the Tennessee Eastman chemical process are used to demonstrate the performance of the proposed FCM-kNN for fault detection and diagnosis.

Instance Selection: A Bayesian Decision Theory Perspective

In this paper, we consider the problem of lacking theoretical foundation and low execution efficiency of the instance selection methods based on the k-nearest neighbour rule when processing large-scale data. We point out that the core idea of these methods can be explained from the perspective of Bayesian decision theory, that is, to find which instances are reducible, irreducible, and deleterious. Then, based on the percolation theory, we establish the relationship between these three types of instances and local homogeneous cluster (i.e., a set of instances with the same labels). Finally, we propose a method based on an accelerated k-means algorithm to construct local homogeneous clusters and remove the superfluous instances. The performance of our method is studied on extensive synthetic and benchmark data sets. Our proposed method can handle large-scale data more effectively than the state-of-the-art instance selection methods. All code and data results are available at https://github.com/CQQXY161120/Instance-Selection.

K-nearest neighbors rule combining prototype selection and local feature weighting for classification

K-Nearest Neighbors (KNN) rule is a simple yet powerful classification technique in machine learning. Nevertheless, it suffers from some drawbacks such as high memory consumption, low time efficiency, class overlapping and difficulty of setting an appropriate K value. In this study, we propose an Improved K-Nearest Neighbor rule combining Prototype Selection and Local Feature Weighting (IKNN_PSLFW) to address the above issues in one framework. Differing from conventional prototype selection, IKNN_PSLFW not only selects the representative instances as prototypes, but also preserves the information of instances that are not selected. To deal with the class overlapping problem, IKNN_PSLFW explores the feature relevance in local regions by assigning different weights to different features. For an instance with unknown class label, IKNN_PSLFW uses three classification rules corresponding to three scenarios, according to the distance between the instance and each prototype, for classification. To evaluate the performance of IKNN_PSLFW, we conduct experimental study on 20 benchmark datasets. The experimental results show that compared with the conventional KNN rule, some state-of-the-art prototype selection methods and other machine learning algorithms, the proposed IKNN_PSLFW achieves promising classification performance with high time efficiency.

A representation coefficient-based k-nearest centroid neighbor classifier

K-nearest neighbor rule (KNN) has been regarded as one of the top 10 methods in the field of data mining. Due to its simplicity and effectiveness, it has been widely studied and applied to various classification tasks. In this article, we develop a novel representation coefficient-based k-nearest centroid neighbor method (RCKNCN), which aims to further improve the classification performance and reduce the method’s sensitivity to the neighborhood size k, especially in the cases of small sample size. Different from existing KNN-based methods, RCKNCN is able to capture both the proximity and the geometry of k-nearest neighbors, and learn to differentiate the contribution of each neighbor to the classification of a testing sample through a linear representation method. Moreover, under the RCKNCN framework, we also propose a novel weighted majority voting algorithm using the representation coefficients associated with individual nearest centroid neighbors, which are deemed to hold more discriminative information of the neighbors. To fully study the classification performance of RCKNCN, we compare it with the state-of-the-art KNN-based methods on many data sets that are widely used in the literature. The extensive experiments demonstrate the effectiveness and robustness of our method in various classification tasks.

Continual learning classification method with the weighted k-nearest neighbor rule for time-varying data space based on the artificial immune system

Supervised learning classification methods play an important role in various fields. However, most of them cannot effectively classify the data from time-varying data spaces, for they lack continual learning ability. Inspired by the intelligent mechanism that memory cells of the biological immune system can evolve with the evolution of invaders, a continual learning classification method with the weighted k-nearest neighbor rule for time-varying data space (WKNN-CLCMTVD) is proposed. Memory cells selected by the weighted k-nearest neighbor rule are used to identify the type of testing data. Memory cells are continuously cultivated, updated, and eliminated through learning testing data to improve the classification ability of WKNN-CLCMTVD. It degenerates into a standard supervised learning classification method when all data independent of time. Take experiments on twenty benchmark datasets, a 2-dimensional synthetic dataset, and XJTU-SY rolling element bearing accelerated life test datasets to evaluate the performance of WKNN-CLCMTVD. Results show that it has better classification ability for time-invariant data and outperforms the other methods for time-varying data space. Results also show that its running speed is far faster than that of other continual learning classification methods.

Quality-related fault diagnosis based onk-nearest neighbor rule for non-linear industrial processes

The fault diagnosis approaches based on k-nearest neighbor rule have been widely researched for industrial processes and achieve excellent performance. However, for quality-related fault diagnosis, the approaches using k-nearest neighbor rule have been still not sufficiently studied. To tackle this problem, in this article, we propose a novel quality-related fault diagnosis framework, which is made up of two parts: fault detection and fault isolation. In the fault detection stage, we innovatively propose a novel non-linear quality-related fault detection method called kernel partial least squares- k-nearest neighbor rule, which organically incorporates k-nearest neighbor rule with kernel partial least squares. Specifically, we first employ kernel partial least squares to establish a non-linear regression model between quality variables and process variables. After that, the statistics and thresholds corresponding to process space and predicted quality space are appropriately designed by adopting k-nearest neighbor rule. In the fault isolation stage, in order to match our proposed non-linear quality-related fault detection method kernel partial least squares- k-nearest neighbor seamlessly, we propose a modified variable contributions by k-nearest neighbor (VCkNN) fault isolation method called modified variable contributions by k-nearest neighbor (MVCkNN), which elaborately introduces the idea of the accumulative relative contribution rate into VC k-nearest neighbor, such that the smearing effect caused by the normal distribution hypothesis of VC k-nearest neighbor can be mitigated effectively. Finally, a widely used numerical example and the Tennessee Eastman process are employed to verify the effectiveness of our proposed approach.

Prototype generation in the string space via approximate median for data reduction in nearest neighbor classification

The k-nearest neighbor (kNN) rule is one of the best-known distance-based classifiers, and is usually associated with high performance and versatility as it requires only the definition of a dissimilarity measure. Nevertheless, kNN is also coupled with low-efficiency levels since, for each new query, the algorithm must carry out an exhaustive search of the training data, and this drawback is much more relevant when considering complex structural representations, such as graphs, trees or strings, owing to the cost of the dissimilarity metrics. This issue has generally been tackled through the use of data reduction (DR) techniques, which reduce the size of the reference set, but the complexity of structural data has historically limited their application in the aforementioned scenarios. A DR algorithm denominated as reduction through homogeneous clusters (RHC) has recently been adapted to string representations but as obtaining the exact median value of a set of string data is known to be computationally difficult, its authors resorted to computing the set-median value. Under the premise that a more exact median value may be beneficial in this context, we, therefore, present a new adaptation of the RHC algorithm for string data, in which an approximate median computation is carried out. The results obtained show significant improvements when compared to those of the set-median version of the algorithm, in terms of both classification performance and reduction rates.

K-relevance vectors: Considering relevancy beside nearness

This study combines two different learning paradigms, k-nearest neighbor (k-NN) rule, as memory-based learning paradigm and relevance vector machines (RVM), as statistical learning paradigm. The purpose is to improve the performance of k-NN rule through selection of important features with sparse Bayesian learning method. This combination is performed in kernel space and is called k-relevance vector (k-RV). The proposed model significantly prunes irrelevant features. Our combination of k-NN and RVM presents a new concept of similarity measurement for k-NN rule, we call it k-relevancy which aims to consider “relevancy” in the feature space beside “nearness” in the input space. We also introduce a new parameter, responsible for early stopping of iterations in RVM that is able to improve the classification accuracy. Intensive experiments are conducted on several classification datasets from University of California Irvine (UCI) repository and two real datasets from computer vision domain. The performance of k-RV is highly competitive compared to a few state-of-the-arts in terms of classification accuracy.

Entropy Based k Nearest Neighbor Pattern Classification (EbkNN): En-route to Achieving a High Accuracy in Breast Cancer Diagnosis

Entropy based k-Nearest Neighbor pattern classification (EbkNN) is a variation of the conventional k-Nearest Neighbor rule of pattern classification, which exclusively optimizes the value of k-neighbors for each test data based on the calculations of entropy. The formula for entropy used in EbkNN is the one that has been defined popularly in information theory for a set of n different types of information (class) attached to a total of m objects (data points) with each object defined by f features. In EbkNN that value of k is chosen for discrimination of given test data for which the entropy is the least non-zero value. Other rules of conventional kNN are retained in EbkNN. It is concluded that EbkNN works best for binary classification. It is computationally prohibitive to use EbkNN for discriminating the data points of the test dataset into number of classes greater than two. The biggest advantage of EbkNN vis-à-vis the conventional kNN is that in one single run of EbkNN algorithm we get optimum classification of test data. But conventional kNN algorithm has to be run separately for each of the selected range of values of k, and then the optimum k to be chosen from amongst them. We also tested our EbkNN method on WDBC (Wisconsin Diagnostic Breast Cancer) dataset. There are 569 instances in this dataset and we made a random choice of first 290 instances as training dataset and the rest 279 instances as test dataset. We got an exceptionally remarkable result with EbkNN method- accuracy close to 100% and better than the ones got by most of the other researchers who worked on WDBC dataset.

A new globally adaptive k-nearest neighbor classifier based on local mean optimization

The k-nearest neighbor (KNN) rule is a simple and effective nonparametric classification algorithm in pattern classification. However, it suffers from several problems such as sensitivity to outliers and inaccurate classification decision rule. Thus, a local mean-based k-nearest neighbor classifier (LMKNN) was proposed to address these problems, which assigns the query sample with a class label based on the closest local mean vector among all classes. It is proven that the LMKNN classifier achieves better classification performance and is more robust to outliers than the classical KNN classifier. Nonetheless, the unreliable nearest neighbor selection rule and single local mean vector strategy in LMKNN classifier severely have negative effect on its classification performance. Considering these problems in LMKNN, we propose a globally adaptive k-nearest neighbor classifier based on local mean optimization, which utilizes the globally adaptive nearest neighbor selection strategy and the implementation of local mean optimization to obtain more convincing and reliable local mean vectors. The corresponding experimental results conducted on twenty real-world datasets demonstrated that the proposed classifier achieves better classification performance and is less sensitive to the neighborhood size $$k$$ compared with other improved KNN-based classification methods.

SMOTE-WENN: Solving class imbalance and small sample problems by oversampling and distance scaling

Many practical applications suffer from imbalanced data classification, in which case the minority class has degraded recognition rate. The primary causes are the sample scarcity of the minority class and the intrinsic complex distribution characteristics of imbalanced datasets. The imbalanced classification problem is more serious on small sample datasets. To solve the problems of small sample and class imbalance, a hybrid resampling method is proposed. The proposed method combines an oversampling approach (synthetic minority oversampling technique, SMOTE) and a novel data cleaning approach (weighted edited nearest neighbor rule, WENN). First, SMOTE generates synthetic minority class examples using linear interpolation. Then, WENN detects and deletes unsafe majority and minority class examples using weighted distance function and k-nearest neighbor (kNN) rule. The weighted distance function scales up a commonly used distance by considering local imbalance and spacial sparsity. Extensive experiments over synthetic and real datasets validate the superiority of the proposed SMOTE-WENN compared with three state-of-the-art resampling methods.

Recognizing distributions rather than goodness-of-fit testing

This article puts forward an idea of recognizing distributions rather than carrying-out classic goodness-of-fit tests (GoFTs). For the purpose of recognizing the k-nearest neighbors (kNN) rule is applied. We focus the reader’s attention on recognizing the normal distribution. The main part of the article is devoted to the computer implementation of a classifier of distributions that involves kNN rule. GoFTs are conservative. Recognizing distributions is exemplified by simulation and real data examples. When the test statistics exceeds relevant critical value then the verdict sounds: there are reasons to reject And what next? Recognizing distributions is the answer to this question.

Using multiple classifier behavior to develop a dynamic outlier ensemble

Outlier ensembles that use more base detectors recently become an attractive approach to solving problems of single detectors. However, existing outlier ensembles often assume that base detectors make independent errors, which is difficult to satisfy in practical applications. To this end, this paper proposes a dynamic outlier ensemble to loose this error independence assumption. In our method, it is desired that the most competent base detector(s) can be singled out by the dynamic selection mechanism for each test pattern. The usage of the concept of multiple classifier behavior (MCB) has two purposes. One is to generate artificial outlier examples used for competence estimates. This strategy is different from other methods since we do not make any assumption regarding the data distribution. On the other hand, MCB is used to refine validation sets initialized by the K-nearest neighbors (KNN) rule. It is desired that objects in the refined validation sets are more representative than those found by KNN. With the refined validation sets, competences of all base detectors will be estimated by a probabilistic method, before which we have transformed outputs of base detectors into a probabilistic form. Finally, a switching mechanism that determines whether one detector should be nominated to make the decision or a fusion method should be applied instead is proposed in order to achieve a robust detection result. We carry out experiments on 20 benchmark data sets to verify the effectiveness of our detection method.

A new locally adaptive [formula omitted]-nearest neighbor algorithm based on discrimination class

The k-nearest neighbor (kNN) rule is a classical non-parametric classification algorithm in pattern recognition, and has been widely used in many fields due to its simplicity, effectiveness and intuitiveness. However, the classification performance of the kNN algorithm suffers from the choice of a fixed and single value of k for all queries in the search stage and the use of simple majority voting rule in the decision stage.In this paper, we propose a new kNN-based algorithm, called locally adaptive k-nearest neighbor algorithm based on discrimination class (DC-LAKNN). In our method, the role of the second majority class in classification is for the first time considered. Firstly, the discrimination classes at different values of k are selected from the majority class and the second majority class in the k-neighborhood of the query. Then, the adaptive k value and the final classification result are obtained according to the quantity and distribution information on the neighbors in the discrimination classes at each value of k.Extensive experiments on eighteen real-world datasets from UCI (University of California, Irvine) Machine Learning Repository and KEEL (Knowledge Extraction based on Evolutionary Learning) Repository show that the DC-LAKNN algorithm achieves better classification performance compared to standard kNN algorithm and nine other state-of-the-art kNN-based algorithms.