Separable Classification Problems Research Articles

Vibration-based incipient surge detection and diagnosis of the centrifugal compressor using adaptive feature fusion and sparse ensemble learning approach

As a critical high-speed rotating machinery, the centrifugal compressor has been widely used in various modern industries. However, it is subject to a potential damaging phenomenon called surge that can be caused by several factors such as unmatched design, improper operation, inlet and outlet blockage, and so on, which thus may result in catastrophic accidents. This paper concerns the incipient surge detection and diagnosis (ISDD) of the centrifugal compressor based on its bearing vibration signals, leveraging adaptive feature fusion and sparse ensemble learning approach to develop a data-driven-based intelligent diagnostic model. Firstly, vibration signals are decomposed by the empirical mode decomposition (EMD) method, and various features can be extracted from the obtained intrinsic mode functions (IMFs) with rich information of the time series. Next, maximum likelihood estimation (MLE) is utilized to compute the intrinsic dimension of the extracted feature vectors, which are then reduced to the corresponding dimensionality adaptively by kernel principal component analysis (kernel PCA). Coping with the multi-dimensional input and nonlinearly separable classification problem, an ensemble learning approach with L1-regularization term is adopted for incipient surge identification of the centrifugal compressor. A novel and hyperparameter-relaxed hybrid algorithm is employed for the sparse ensemble learning procedure, which guarantees transcendental accuracy and preeminent generalization capability while ensuring high robustness and sparsity of the intelligent model. Finally, in addition to diagnostic tasks of the centrifugal compressor datasets under different working conditions, the ISDD performance of the proposed method in imbalanced sample scenarios, varying working conditions and additional noise interference circumstances is comprehensively investigated, and the overall success indicates the robustness and superiority of the proposed method compared with other existing approaches.

In-depth analysis of SVM kernel learning and its components

The performance of support vector machines in nonlinearly separable classification problems strongly relies on the kernel function. Toward an automatic machine learning approach for this technique, many research outputs have been produced dealing with the challenge of automatic learning of good-performing kernels for support vector machines. However, these works have been carried out without a thorough analysis of the set of components that influence the behavior of support vector machines and their interaction with the kernel. These components are related in an intricate way and it is difficult to provide a comprehensible analysis of their joint effect. In this paper, we try to fill this gap introducing the necessary steps in order to understand these interactions and provide clues for the research community to know where to place the emphasis. First of all, we identify all the factors that affect the final performance of support vector machines in relation to the elicitation of kernels. Next, we analyze the factors independently or in pairs and study the influence each component has on the final classification performance, providing recommendations and insights into the kernel setting for support vector machines.

Hash Transformation and Machine Learning-Based Decision-Making Classifier Improved the Accuracy Rate of Automated Parkinson's Disease Screening.

Digitalized hand-drawn pattern is a noninvasive and reproducible assistive manner to obtain hand actions and motions for evaluating functional tremors and upper-limb movement disorders. In this study, spirals and straight lines in polar coordinates are used to extract polar expression features such as the key parameters deviation (cm) and accumulation angle (rad). These parameters are quantitative manner to scale the variations of functional tremors in normal control subjects and patients with Parkinson's disease (PD) and essential tremor (ET). However, difficulty arises in using nonlinear polar expression features in the two-dimensional feature space to separate normal control subjects from those with PD and ET. To solve the nonlinear separable classification problem, hash transformation is used to map polar expression features to a high-dimensional space using hash weighing function and modulo operation. Then, a machine learning method, such as the generalized regression neural network (GRNN), is implemented to train a decision-making classifier using the particle swarm optimization (PSO) algorithm for possible class assessment. With the enrolled data from 50 subjects, the fivefold cross validation, mean true positive, mean true negative, and mean hit rates of 98.93%, 98.96%, and 98.93%, respectively, are obtained to quantify the performance of the proposed decision-making classifier to identify normal controls and subjects with PD or ET. The experimental results indicate that the proposed screening model can improve the accuracy rate compared with the conventional machine learning classifier.

Competitive Cross-Entropy Loss: A Study on Training Single-Layer Neural Networks for Solving Nonlinearly Separable Classification Problems

After Minsky and Papert (Perceptrons, MIT Press, Cambridge, 1969) showed the inability of perceptrons in solving nonlinearly separable problems, for several decades people misinterpreted it as an inherent weakness that is common to all single-layer neural networks. The introduction of the backpropagation algorithm reinforced this misinterpretation as its success in solving nonlinearly separable problems passed through the training of multilayer neural networks. Recently, Conaway and Kurtz (Neural Comput 29(3):861–866, 2017) proposed a single-layer network in which the number of output units for each class is the same as input units and showed that it could solve some nonlinearly separable problems. They used the MSE (Mean Square Error) between the input units and the output units of the actual class as the objective function for training the network. They showed that their method could solve the XOR and M&S’81 problems, but it could not do any better than random guessing on the 3-bit parity problem. In this paper, we use a soft competitive approach to generalize the CE (Cross-Entropy) loss, which is a widely accepted criterion for multiclass classification, to networks that have several output units for each class, calling the resulting measure the CCE (Competitive cross-entropy) loss. In contrast to Conaway and Kurtz (2017), in our method, the number of output units for each class can be chosen arbitrarily. We show that the proposed method can successfully solve the 3-bit parity problem, in addition to the XOR and M&S’81 problems. Furthermore, we perform experiments on several datasets for multiclass classification, comparing a single-layer network trained with the proposed CCE loss against LVQ, linear SVM, a single-layer network trained with the CE loss, and the method of Conaway and Kurtz (2017). The results show that the CCE loss performs remarkably better than existing algorithms for training single-layer neural networks.

Incremental p-margin algorithm for classification with arbitrary norm

This paper presents a new algorithm to approximate large margin solutions in binary classification problems with arbitrary q-norm or p-margin, where p and q are Holder conjugates. We begin by presenting the online fixed p-margin perceptron algorithm (FMPp) that solves linearly separable classification problems in primal variables and consists of a generalization of the fixed margin perceptron algorithm (FMP). This algorithm is combined with an incremental margin strategy called IMAp, which computes an approximation of the maximal p-margin. To achieve this goal, IMAp executes FMPp several times with increasing p-margin values. One of the main advantages of this approach is its flexibility, which allows the use of different p-norms in the same primal formulation. For non-linearly separable problems, FMPp can be used with a soft margin in primal variables. The incremental learning strategy always guarantees a good approximation of the optimal p-margin and avoids the use of linear or higher order programming methods. IMAp was tested in different datasets obtaining similar results when compared to classical L1 and L∞ linear programming formulations. Also, the algorithm was compared to ALMAp and presents superior results.

Improved multi-kernel classification machine with Nyström approximation technique and Universum data

Universum learning can reflect priori knowledge about application domain and improve classification performances. The kernelized modification of Ho–Kashyap algorithm with squared approximation of the misclassification errors (KMHKS) is an effective learning machine for nonlinearly separable classification problems. While KMHKS only adopts one kernel function, so a multi-kernel classification machine with reduced complexity named Nyström approximation matrix with multiple KMHKSs (NMKMHKS) has been developed. But NMKMHKS has to initialize many parameters and it has not an ability to process noises well. To this end, some scholars propose an improved multi-kernel classification machine with Nyström approximation technique (INMKMHKS). INMKMHKS is based on a new way of generating kernel functions and a new Nyström approximation technique. Related experiments have validated that INMKMHKS possesses five advantages: (1) avoiding the problem of setting too many parameters; (2) keeping comparable space and computational complexities after comparing with NMKMHKS; (3) having a tighter generalization risk bound in terms of Rademacher complexity analysis; (4) possessing an ability to process noises and practical images; (5) a superior recognition can be gotten in a strong correlation between multiple used kernels, which can give a guide advice for choosing kernels. But for traditional multiple kernel learning (MKL), many classification machines including INMKMHKS focus on MKL optimizations, for example, the optimization of model for a MKL. It is difficult for us to find or create a new optimization way now. Indeed, if one pays more attention to data themselves, performance of MKL will also have an improvement. This paper adopts INMKMHKS as a basic learning machine and focuses on data themselves, then it introduces Universum learning into the procedure of INMKMHKS and proposes Universum-based INMKMHKS (Uni-INMKMHKS). The motivation of Uni-INMKMHKS is that it can design a learning machine from data perspective and avoid paying too much attention to MKL optimization which is difficult for scholars to some extent. The contribution of Uni-INMKMHKS is that it has a better recognition than INMKMHKS in average with Universum learning used and inherits the advantages of INMKMHKS simultaneously.

Multi-input distributed classifiers for synthetic genetic circuits.

For practical construction of complex synthetic genetic networks able to perform elaborate functions it is important to have a pool of relatively simple modules with different functionality which can be compounded together. To complement engineering of very different existing synthetic genetic devices such as switches, oscillators or logical gates, we propose and develop here a design of synthetic multi-input classifier based on a recently introduced distributed classifier concept. A heterogeneous population of cells acts as a single classifier, whose output is obtained by summarizing the outputs of individual cells. The learning ability is achieved by pruning the population, instead of tuning parameters of an individual cell. The present paper is focused on evaluating two possible schemes of multi-input gene classifier circuits. We demonstrate their suitability for implementing a multi-input distributed classifier capable of separating data which are inseparable for single-input classifiers, and characterize performance of the classifiers by analytical and numerical results. The simpler scheme implements a linear classifier in a single cell and is targeted at separable classification problems with simple class borders. A hard learning strategy is used to train a distributed classifier by removing from the population any cell answering incorrectly to at least one training example. The other scheme implements a circuit with a bell-shaped response in a single cell to allow potentially arbitrary shape of the classification border in the input space of a distributed classifier. Inseparable classification problems are addressed using soft learning strategy, characterized by probabilistic decision to keep or discard a cell at each training iteration. We expect that our classifier design contributes to the development of robust and predictable synthetic biosensors, which have the potential to affect applications in a lot of fields, including that of medicine and industry.

Improved multi-kernel classification machine with Nyström approximation technique

Kernelized modification of Ho–Kashyap algorithm with squared approximation of the misclassification errors (KMHKS) is an effective algorithm for nonlinearly separable classification problems. While KMHKS only adopts one kernel function. So a multi-kernel classification machine with reduced complexity named Nyström approximation matrix with Multiple KMHKSs (NMKMHKS) has been developed. But NMKMHKS has to initialize many parameters and has not an ability to deal with noise well. To this end, we propose an improved multi-kernel classification machine with Nyström approximation technique (INMKMHKS). INMKMHKS is based on a new way of generating kernel functions and a new Nyström approximation technique. The contributions of INMKMHKS are that (1) avoiding the problem of setting too many parameters; (2) keeping comparable space and computational complexities after comparing with NMKMHKS; (3) having a tighter generalization risk bound in terms of Rademacher complexity analysis; (4) having a better recognition than NMKMHKS on average; (5) possessing an ability to deal with noise and practical images.

An Approach to Improve Functional Link Neural Network Training Using Modified Artificial Bee Colony for Classification Task

Classification is one of the most frequent studies in the area of Artificial Neural Network (ANNs) . One of the best known types of ANNs is the Multilayer Perceptron (MLP). However, MLP usually requires a rather large amount of available measures in order to achieve good classification accuracy. To overcome this, a Functional Link Neural Networks (FLNN), which has single layer of trainable connection weight is used. The standard method for tuning the weight in FLNN is using a Backpropagation (BP) learning algorithm. Still, BP-learning algorithm has difficulties such as trapping in local optima and slow convergence especially for solving non-linearly separable classification problems. In this paper, a modified Artificial Bee Colony (mABC) is used to recover the BP drawbacks. With modifications on the employed bee’s exploitation phase, the implementation of the mABC as a learning scheme for FLNN has given a better accuracy result for the classification tasks.

CONSTRUCTIVE RECURSIVE DETERMINISTIC PERCEPTRON NEURAL NETWORKS WITH GENETIC ALGORITHMS

The recursive deterministic perceptron (RDP) is a generalization of the single layer perceptron neural network. This neural network can separate, in a deterministic manner, any classification problem (linearly separable or not). It relies on the principle that in any nonlinearly separable (NLS) two-class classification problem, a linearly separable (LS) subset of one or more points belonging to one of the two classes can always be found. Small network topologies can be obtained when the LS subsets are of maximum cardinality. This is referred to as the problem of maximum separability and has been proven to be NP-Complete. Evolutionary computing techniques are applied to handle this problem in a more efficient way than the standard approaches in terms of complexity. These techniques enhance the RDP training in terms of speed of conversion and level of generalization. They provide an alternative to tackle large classification problems which is otherwise not feasible with the algorithmic versions of the RDP training methods.

Finding potential support vectors in separable classification problems.

This paper considers the classification problem using support vector (SV) machines and investigates how to maximally reduce the size of the training set without losing information. Under separable data set assumptions, we derive the exact conditions stating which observations can be discarded without diminishing the overall information content. For this purpose, we introduce the concept of potential SVs, i.e., those data that can become SVs when future data become available. To complement this, we also characterize the set of discardable vectors (DVs), i.e., those data that, given the current data set, can never become SVs. Thus, these vectors are useless for future training purposes and can eventually be removed without loss of information. Then, we provide an efficient algorithm based on linear programming that returns the potential and DVs by constructing a simplex tableau. Finally, we compare it with alternative algorithms available in the literature on some synthetic data as well as on data sets from standard repositories.

Adaptive Iterative Approach Classifying Linearly and Quadratically Separable Sets

This paper presents a fast adaptive iterative algorithm to solve linearly separable classification problems in Rn. In each iteration step, a subset of the sampling data (n-points) is adaptively chosen and a hyperplane is constructed such that it separates the n-points at a margin e and it best classifies the remaining points. The classification problem is formulated and three different algorithms are presented. Further, the algorithm is extended to solve quadratically separable classification problems. The idea is based on mapping the physical space to another larger one where the problem becomes linearly separable. Various numerical illustrations and comparisons with other classification algorithms using benchmark datasets are presented. Numerical results show that few iteration steps are sufficient for convergence and demonstrate that the proposed approach achieves 100 % correctness for linearly separable datasets, whereas other learning techniques may fail to achieve such percentage.

Linear separability and classification complexity

We study the relationship between linear separability and the level of complexity of classification data sets. Linearly separable classification problems are generally easier to solve than non linearly separable ones. This suggests a strong correlation between linear separability and classification complexity. We propose a novel and simple method for quantifying the complexity of the classification problem. The method, which is shown below, reduces any two class classification problem to a sequence of linearly separable steps. The number of such reduction steps could be viewed as measuring the degree of non-separability and hence the complexity of the problem. This quantification in turn can be used as a measure for the complexity of classification data sets. Results obtained using several benchmarks are provided.

Geometric algorithms to large margin classifier based on affine hulls.

The geometric framework for binary data classification problems provides an intuitive foundation for the comprehension and application of geometric optimization algorithms, leading to practical solutions of real-world classification problems. In this paper, some theoretical results on the candidate extreme points of the notion of reduced affine hull (RAH) are introduced. These results allow the existing nearest point algorithms to be directly applied to solve both separable and inseparable classification problems based on RAHs successfully and efficiently. As the practical applications of the new theoretical results, the popular Gilbert-Schlesinger-Kozinec and Mitchell-Dem'yanov-Malozemov algorithms are presented to solve binary classification problems in the context of the RAH framework. The theoretical analysis and some experiments show that the proposed methods successfully achieve significant performance.

Linearly and Quadratically Separable Classifiers Using Adaptive Approach

This paper presents a fast adaptive iterative algorithm to solve linearly separable classification problems in Rn. In each iteration, a subset of the sampling data (n-points) is adaptively chosen and a hyperplane is constructed such that it separates the n-points at a margin E and it best classifies the remaining points. The classification problem is formulated and three different algorithms are presented. Numerical results show that few iterations are sufficient for convergence. Further, the algorithm is extended to solve quadratically separable classification problems. The basic idea is based on mapping the physical space to another larger one where the problem becomes linearly separable.

A hybrid radial basis function and data envelopment analysis neural network for classification

We propose a hybrid radial basis function network-data envelopment analysis (RBFN-DEA) neural network for classification problems. The procedure uses the radial basis function to map low dimensional input data from input space ℜ to a high dimensional ℜ + feature space where DEA can be used to learn the classification function. Using simulated datasets for a non-linearly separable binary classification problem, we illustrate how the RBFN-DEA neural network can be used to solve it. We also show how asymmetric misclassification costs can be incorporated in the hybrid RBFN-DEA model. Our preliminary experiments comparing the RBFN-DEA with feed forward and probabilistic neural networks show that the RBFN-DEA fares very well.

Kernel-based multiple criteria linear programming classifier

Abstract This paper proposed a novel classification model which introduced the kernel function into the original Multiple Criteria Linear Programming (MCLP) model. MCLP model is used as a classification method which can only solve linear separable problems in data mining. However, the proposed kernel-based MCLP model can deal with non-linear cases. Meanwhile, unlike some other complicated models, this model is effective and easy to understand. A couple of experimental tests were conducted to evaluate the performance of the proposed kernel-based MCLP model compared with the existing methods: original MCLP and SVM. The results show that kernel-based MCLP model is a competitive method in dealing with nonlinear separable classification problems.

SoftDoubleMaxMinOver: Perceptron-Like Training of Support Vector Machines

The well-known MinOver algorithm is a slight modification of the perceptron algorithm and provides the maximum-margin classifier without a bias in linearly separable two-class classification problems. DoubleMinOver as an extension of MinOver, which now includes a bias, is introduced. An O(t(-1)) convergence is shown, where t is the number of learning steps. The computational effort per step increases only linearly with the number of patterns. In its formulation with kernels, selected training patterns have to be stored. A drawback of MinOver and DoubleMinOver is that this set of patterns does not consist of support vectors only. DoubleMaxMinOver, as an extension of DoubleMinOver, overcomes this drawback by selectively forgetting all nonsupport vectors after a finite number of training steps. It is shown how this iterative procedure that is still very similar to the perceptron algorithm can be extended to classification with soft margins and be used for training least squares support vector machines (SVMs). On benchmarks, the SoftDoubleMaxMinOver algorithm achieves the same performance as standard SVM software.

A Novel Geometric Approach to Binary Classification Based on Scaled Convex Hulls

Geometric methods are very intuitive and provide a theoretical foundation to many optimization problems in the fields of pattern recognition and machine learning. In this brief, the notion of scaled convex hull (SCH) is defined and a set of theoretical results are exploited to support it. These results allow the existing nearest point algorithms to be directly applied to solve both the separable and nonseparable classification problems successfully and efficiently. Then, the popular S-K algorithm has been presented to solve the nonseparable problems in the context of the SCH framework. The theoretical analysis and some experiments show that the proposed method may achieve better performance than the state-of-the-art methods in terms of the number of kernel evaluations and the execution time.

Dual unification of bi-class support vector machine formulations

Support vector machine (SVM) theory was originally developed on the basis of a linearly separable binary classification problem, and other approaches have been later introduced for this problem. In this paper it is demonstrated that all these approaches admit the same dual problem formulation in the linearly separable case and that all the solutions are equivalent. For the non-linearly separable case, all the approaches can also be formulated as a unique dual optimization problem, however, their solutions are not equivalent. Discussions and remarks in the article point to an in-depth comparison between SVM formulations and associated parameters.