Large-scale Classification Problems Research Articles

Self-organizing lightweight correlation-aware fuzzy broad learning system for high-dimensional large-scale classification problems

Self-organizing lightweight correlation-aware fuzzy broad learning system for high-dimensional large-scale classification problems

A new fast ADMM for kernelless SVM classifier with truncated fraction loss

Support vector machine (SVM) is one of well-known supervised machine learning classifier and is used widely in image classification, pattern recognition, disease diagnosis, etc. However, the high computational complexity is a major issue for large-scale classification problems. To address this issue, we propose a new truncated fraction loss function to construct a new sparse and robust SVM model (Ltf-SVM). By newly developed support vectors and a working set based on the proximal operator of truncated fraction loss, we prove the sparsity and robustness to outliers of Ltf-SVM from a theoretical perspective. When it comes to solve Ltf-SVM problems, we develop an alternating direction method of multipliers with a working set for addressing Ltf-SVM. The proposed algorithm is proven to converge globally and enjoys very low computational complexity. Moreover, numerical experiments verify its superior performance with respect to its high accuracy and high computational speed in comparison with other leading solvers.

Fast SVM classifier for large-scale classification problems

Support vector machines (SVM), as one of effective and popular classification tools, have been widely applied in various fields. However, they may incur prohibitive computational costs when solving large-scale classification problems. To address this problem, we construct a new fast SVM with a truncated squared hinge loss (dubbed as Lts-SVM). We begin by developing an optimality theory of the nonconvex and nonsmooth Lts-SVM, which makes it convenient for us to investigate the support vectors and working set of Lts-SVM. Based on this, we propose a new and effective global convergence algorithm to address the Lts-SVM. This method is found to enjoy a tremendously low computational complexity, which makes sufficiently decreasing the demand for extremely large-scale computation possible. Numerical comparisons with eight other solvers show that our proposed algorithm achieves excellent performance on large-scale classification problems with regard to shorter computational times, more desirable accuracy levels, fewer support vectors and more robust to outliers.

Large-Scale Fuzzy Least Squares Twin SVMs for Class Imbalance Learning

Twin support vector machines (TSVMs) have been successfully employed for binary classification problems. With the advent of machine learning algorithms, data have proliferated and there is a need to handle or process large-scale data. TSVMs are not successful in handling large-scale data due to the following: 1) the optimization problem solved in the TSVM needs to calculate large matrix inverses, which makes it an ineffective choice for large-scale problems; 2) the empirical risk minimization principle is employed in the TSVM and, hence, may suffer due to overfitting; and 3) the Wolfe dual of TSVM formulation involves positive-semidefinite matrices, and hence, singularity issues need to be resolved manually. Keeping in view the aforementioned shortcomings, in this article, we propose a novel large-scale fuzzy least squares TSVM for class imbalance learning (LS-FLSTSVM-CIL). We formulate the LS-FLSTSVM-CIL such that the proposed optimization problem ensures that: 1) no matrix inversion is involved in the proposed LS-FLSTSVM-CIL formulation, which makes it an efficient choice for large-scale problems; 2) the structural risk minimization principle is implemented, which avoids the issues of overfitting and results in better performance; and 3) the Wolfe dual formulation of the proposed LS-FLSTSVM-CIL model involves positive-definite matrices. In addition, to resolve the issues of class imbalance, we assign fuzzy weights in the proposed LS-FLSTSVM-CIL to avoid bias in dominating the samples of class imbalance problems. To make it more feasible for large-scale problems, we use an iterative procedure known as the sequential minimization principle to solve the objective function of the proposed LS-FLSTSVM-CIL model. From the experimental results, one can see that the proposed LS-FLSTSVM-CIL demonstrates superior performance in comparison to baseline classifiers. To demonstrate the feasibility of the proposed LS-FLSTSVM-CIL on large-scale classification problems, we evaluate the classification models on the large-scale normally distributed clustered (NDC) dataset. To demonstrate the practical applications of the proposed LS-FLSTSVM-CIL model, we evaluate it for the diagnosis of Alzheimer’s disease and breast cancer disease. Evaluation on NDC datasets shows that the proposed LS-FLSTSVM-CIL has feasibility in large-scale problems as it is fast in comparison to the baseline classifiers.

TSVM-M[formula omitted]: Twin support vector machine based on multi-order moment matching for large-scale multi-class classification

For multi-class classification, many existing methods, such as multiple weighted linear loss twin support vector machine (MWLTSVM), construct multiple decision hyperplanes by minimizing the positive points loss’s first-order moment (mean), which may lead to sensitivity to outliers. Also, when faced with a large-scale classification problem, how to speed up the process of solving the optimization model is also a challenge. An alternative is to use rectangular kernel technology (RKT) to reduce computational complexity. However, RKT is based on the uniform point selection method, which can be proven to be ineffective in improving classifier performance. To address these problems, a novel classifier under the structure of “one-versus-rest” for multi-class classification is proposed in this paper, named twin support vector machine based on multi-order moment matching (TSVM-M3). When constructing the decision hyperplanes, TSVM-M3 takes the first-order and second-order moments (mean and variance) of positive points loss into consideration and implements this by introducing an adjusting factor into the objective function. A theoretical analysis of the robustness of the proposed TSVM-M3 is also provided. Meanwhile, a novel RKT based on the density-dependent data selection method is proposed for large-scale classification. We demonstrate that the proposed RKT can benefit from reducing modeling error. Experimental results show the effectiveness of the proposed TSVM-M3.

Vehicle and Person Re-Identification With Support Neighbor Loss.

One of the key tasks for an intelligent visual surveillance system is to automatically re-identify objects of interest, e.g., persons or vehicles, from nonoverlapping camera views. This demand incurs the vast investigation of person re-identification (re-ID) and vehicle re-ID techniques, especially those deep learning-based ones. While most recent algorithms focus on designing new convolutional neural networks, less attention is paid to the loss functions, which are of vital roles as well. Triplet loss and softmax loss are the two losses that are extensively used, both of which, however, have limitations. Triplet loss optimizes the model to produce features with which samples from the same class have higher similarity than those from different classes. The problem of triplet loss is that the number of triplets to be constructed grows cubically with training samples, which causes scalability issue, unstable performance, and slow convergence. Softmax loss has favorable scalable property and is widely used for large-scale classification problems. However, since Softmax loss only aims to separate well training classes, its performance for re-ID tasks is not desirable because the model is tested to measure the similarity of samples from unseen classes. We propose the support neighbor (SN) loss, which avoids the limitations of the abovementioned two losses. Unlike triplet loss that is calculated based on triplets, SN loss is derived from K -nearest neighbors (SNs) of anchor samples. The SNs of an anchor are unique, containing more valuable contextual information and neighborhood structure of the anchor, and thus contribute to more stable performance and reliable embedding from image space to feature space. Based on the SNs, a softmax-like separation term and a squeeze term are proposed, which encourage interclass separation and intraclass compactness, respectively. Experiments show that SN loss surpasses triplet and softmax losses with the same backbone network and reaches the state-of-the-art performance for both person and vehicle re-ID using a ResNet50 backbone when combined with training tricks.

Validation of Large-Scale Classification Problem in Dendritic Neuron Model Using Particle Antagonism Mechanism

With the characteristics of simple structure and low cost, the dendritic neuron model (DNM) is used as a neuron model to solve complex problems such as nonlinear problems for achieving high-precision models. Although the DNM obtains higher accuracy and effectiveness than the middle layer of the multilayer perceptron in small-scale classification problems, there are no examples that apply it to large-scale classification problems. To achieve better performance for solving practical problems, an approximate Newton-type method-neural network with random weights for the comparison; and three learning algorithms including back-propagation (BP), biogeography-based optimization (BBO), and a competitive swarm optimizer (CSO) are used in the DNM in this experiment. Moreover, three classification problems are solved by using the above learning algorithms to verify their precision and effectiveness in large-scale classification problems. As a consequence, in the case of execution time, DNM + BP is the optimum; DNM + CSO is the best in terms of both accuracy stability and execution time; and considering the stability of comprehensive performance and the convergence rate, DNM + BBO is a wise choice.

Sketch discriminatively regularized online gradient descent classification

Online learning represents an important family of efficient and scalable algorithms for large-scale classification problems. Many of them are linear with fast computational speed, but when faced with complex classification, they more likely have low accuracies. In order to improve accuracies, kernel trick is applied, however, it often brings high computational cost. In fact, discriminative information is vital in classification which is still not fully utilized in these algorithms. In this paper, we proposed a novel online linear method, called Sketch Discriminatively Regularized Online Gradient Descent Classification (SDROGD). In order to exploit inter-class separability and intra-class compactness, SDROGD utilizes a matrix to characterize the discriminative information and embeds it directly into a new regularization term. This matrix can be updated by the sketch technique in an online manner. After applying a simple but effective optimization, we show that SDROGD has a good time complexity bound, which is linear with the feature dimension or the number of samples. Experimental results on both toy and real-world datasets demonstrate that SDROGD has not only faster computational speed but also much better classification accuracies than some related kernelized algorithms.

Solving Large Scale Classification Problems with Stochastic Based Optimization

Abstract Development of efficient neural network classifier topology involves preprocessing of training data to reduce noise and discrepancies, utilizing a number of hidden layers, choice of correct activation functions etc. In this research we propose using a combination of first order Stochastic Gradient Descent (SGD) optimizers for Artificial Neural Networks (ANNs) with Deep Learning (DL) and Deep Learning Multilayered Perceptron DLMLP with Back Propagation (BP) for classification of multiclass real datasets with large dimensions. DLMLP topologies require varying the number of hidden layers and their neurons, the development of specific activation and loss functions, batch processing, normalization, implementation of different gradient methods, approaches to avoid overfitting/underfitting as well as evaluation of training and classification results. Our focus is on optimization of stochastic objectives and parameters as well as implementation of different activation functions. K-fold cross validation is utilized to compare the performance of each model with the respective set of parameters. Experiments shown here include Cover Type, Mushrooms, EEG and Plastic Detection datasets where the loss and accuracy metrics are compiled and visualized.

Weighted Linear Loss Projection Twin Support Vector Machine for Pattern Classification

Based on the recently proposed projection twin support vector machine (PTSVM) and least squares projection twin support vector machine (LSPTSVM), in this paper, we propose a weighted linear loss projection twin support vector machine, namely WLPTSVM for short. By introducing the weighted linear loss function, the proposed WLPTSVM not only solves systems of linear equations with lower computational cost but also obtains comparable classification accuracy. In addition, it is able to dispose of large scale classification problems efficiently without any extra external optimizers. The experiments conducted on synthetic and several benchmark datasets illustrate the effectiveness of our WLPTSVM.

Random Projection for Linear 2-norm Support Vector Machine

The linear 2-norm Support Vector Machine (L2-SVM) builds a hyper-plane which maximizes the 2-norm soft margin. Random projection is a new oblivious feature extraction and dimension reduction method. Both techniques are widely applied in compressed sensing, texture classification, face recognition, and so on. This paper find that random projection can be applied to any input matrix of L2-SVM. Furthermore, we proves that the geometric margin and the minimum enclosing ball in the projected space are almost unchanged with high probability compared with those in the original space. The result is demonstrated by experiments on synthetic and real data. Computational experiments also show that the proposed random projection for L2-SVM has a better performance than random projection for L1-SVM. Besides, the proposed model can solve the large scale classification problems more effectively and efficiently. Moreover, the proposed method is compared with Principal Component Analysis and is applied in Corel images classification problems.

Lossless Pruned Naive Bayes for Big Data Classifications

Abstract In a fast growing big data era, volume and varieties of data processed in Internet applications drastically increase. Real-world search engines commonly use text classifiers with thousands of classes to improve relevance or data quality. These large scale classification problems lead to severe runtime performance challenges, so practitioners often resort to fast approximation techniques. However, the increase in classification speed comes at a cost, as approximations are lossy, mis-assigning classes relative to the original reference classification algorithm. To address this problem, we introduce a Lossless Pruned Naive Bayes (LPNB) classification algorithm tailored to real-world, big data applications with thousands of classes. LPNB achieves significant speed-ups by drawing on Information Retrieval (IR) techniques for efficient posting list traversal and pruning. We show empirically that LPNB can classify text up to eleven times faster than standard Naive Bayes on a real-world data set with 7205 classes, with larger gains extrapolated for larger taxonomies. In practice, the achieved acceleration is significant as it can greatly cut required computation time. In addition, it is lossless: the output is identical to standard Naive Bayes, in contrast to extant techniques such as hierarchical classification. The acceleration does not rely on the taxonomy structure, and it can be used for both hierarchical and flat taxonomies.

A new MapReduce associative classifier based on a new storage format for large-scale imbalanced data

The process of knowledge discovery from big and high dimensional datasets has become a popular research topic. The classification problem is a key task in bioinformatics, business intelligence, decision science, astronomy, physics, etc. Building associative classifiers has been a notable research interest in recent years because of their superior accuracy. In associative classifiers, using under-sampling or over-sampling methods for imbalanced big datasets reduces accuracy or increases running time, respectively. Hence, there is a significant need to create efficient associative classifiers for imbalanced big data problems. These classifiers should be able to handle challenges such as memory usage, running time and efficiently exploring the search space. To this end, efficient calculation of measures is a primary objective for associative classifiers. In this paper, we propose a new efficient associative classifier for big imbalanced datasets. The proposed method is based on Rare-PEARs (a multi-objective evolutionary algorithm that efficiently discovers rare and reliable association rules) and is able to evaluate rules in a distributed manner by using a new storing data format. This format simplifies measures calculation and is fully compatible with the MapReduce programming model. We have applied the proposed method (RPII) on a well-known big dataset (ECBDL’14) and have compared our results with seven other learning methods. The experimental results show that RPII outperform other methods in sensitivity and final score measures (the values of sensitivity and final score measures were approximately 0.74 and 0.54 respectively). The results demonstrate that the proposed method is a good candidate for large-scale classification problems; furthermore, it achieves reasonable execution time when the target platform is a typical computer clusters.

Insensitive stochastic gradient twin support vector machines for large scale problems

Within the large scale classification problem, the stochastic gradient descent method called PEGASOS has been successfully applied to support vector machines (SVMs). In this paper, we propose a stochastic gradient twin support vector machine (SGTSVM) based on the twin support vector machine (TWSVM). Compared to PEGASOS, our method is insensitive to stochastic sampling. Furthermore, we prove the convergence of SGTSVM and the approximation between TWSVM and SGTSVM under uniform sampling, whereas PEGASOS is almost surely convergent and only has an opportunity to obtain an approximation to SVM. In addition, we extend SGTSVM to nonlinear classification problems via a kernel trick. Experiments on artificial and publicly available datasets show that our method has stable performance and can handle large scale problems easily.

Sparse feature kernel multi-criteria linear programming classifier

Abstract For a classification problem, the nonlinearly separable case can be transformed into a linearly separable one by using the kernel trick, However, introducing the single kernel function to a classifier can cause the difficulty of feature selection and it reduces the model interpretability, which is very important for some practical applications. Moreover, multi-kernel learning used for improving the predictive accuracy can lead to the poor efficiency and interpretability, especially for large-scale high-dimensional classification problems. In this paper, we proposed a novel sparse feature kernel multi-criteria linear programming classifier (SFK-MCLPC), the two-stage classifier model employs new row and column kernel matrices of different feature kernels to iteratively solve two associative linear programming problems so as to identify important features from data in the process of classification. Based on the ten real-world data sets, the experimental results and comparison with multi-criteria linear and quadratic programming classifiers (MCLPC, MCQPC), support vector classifier (SVC), and multiple kernel learning SVC (MKL-SVC) have shown that the proposed SFK-MCLPC can enhance the predictive accuracy of different classes, the efficiency of classification, the interpretability of the pattern analysis, and the generalization ability of predicting the classes of unseen instances by selecting the relevant and important features.

An improved brain emotional learning algorithm for accurate and efficient data analysis

To overcome the deficiencies of high computational complexity and low convergence speed in traditional neural networks, a novel bio-inspired machine learning algorithm named brain emotional learning (BEL) is introduced. BEL mimics the emotional learning mechanism in brain which has the superior features of fast learning and quick reacting. To further improve the performance of BEL in data analysis, genetic algorithm (GA) is adopted for optimally tuning the weights and biases of amygdala and orbitofrontal cortex in BEL neural network. The integrated algorithm named GA-BEL combines the advantages of the fast learning of BEL, and the global optimum solution of GA. GA-BEL has been tested on a real-world chaotic time series of geomagnetic activity index for prediction, eight benchmark datasets of university California at Irvine (UCI) and a functional magnetic resonance imaging (fMRI) dataset for classifications. The comparisons of experimental results have shown that the proposed GA-BEL algorithm is more accurate than the original BEL in prediction, and more effective when dealing with large-scale classification problems. Further, it outperforms most other traditional algorithms in terms of accuracy and execution speed in both prediction and classification applications.

A Game-Theoretic Approach to Design Secure and Resilient Distributed Support Vector Machines.

Distributed support vector machines (DSVMs) have been developed to solve large-scale classification problems in networked systems with a large number of sensors and control units. However, the systems become more vulnerable, as detection and defense are increasingly difficult and expensive. This paper aims to develop secure and resilient DSVM algorithms under adversarial environments in which an attacker can manipulate the training data to achieve his objective. We establish a game-theoretic framework to capture the conflicting interests between an adversary and a set of distributed data processing units. The Nash equilibrium of the game allows predicting the outcome of learning algorithms in adversarial environments and enhancing the resilience of the machine learning through dynamic distributed learning algorithms. We prove that the convergence of the distributed algorithm is guaranteed without assumptions on the training data or network topologies. Numerical experiments are conducted to corroborate the results. We show that the network topology plays an important role in the security of DSVM. Networks with fewer nodes and higher average degrees are more secure. Moreover, a balanced network is found to be less vulnerable to attacks.

Incremental conic functions algorithm for large scale classification problems

In order to cope with classification problems involving large datasets, we propose a new mathematical programming algorithm by extending the clustering based polyhedral conic functions approach. Despite the high classification efficiency of polyhedral conic functions, the realization previously required a nested implementation of k-means and conic function generation, which has a computational load related to the number of data points. In the proposed algorithm, an efficient data reduction method is employed to the k-means phase prior to the conic function generation step. The new method not only improves the computational efficiency of the successful conic function classifier, but also helps avoiding model over-fitting by giving fewer (but more representative) conic functions.

Large-scale structural learning and predicting via hashing approximation

By combining the structural information with nonparallel support vector machine, structural nonparallel support vector machine (SNPSVM) can fully exploit prior knowledge to directly improve the algorithm’s generalization capacity. However, the scalability issue how to train SNPSVM efficiently on data with huge dimensions has not been studied. In this paper, we integrate linear SNPSVM with b-bit minwise hashing scheme to speedup the training phase for large-scale and high-dimensional statistical learning, and then we address the problem of speeding-up its prediction phase via locality-sensitive hashing. For one-against-one multi-class classification problems, a two-stage strategy is put forward: a series of hash-based classifiers are built in order to approximate the exact results and filter the hypothesis space in the first stage and then the classification can be refined by solving a multi-class SNPSVM on the remaining classes in the second stage. The proposed method can deal with large-scale classification problems with a huge number of features. Experimental results on two large-scale datasets (i.e., news20 and webspam) demonstrate the efficiency of structural learning via b-bit minwise hashing. Experimental results on the ImageNet-BOF dataset, and several large-scale UCI datasets show that the proposed hash-based prediction can be more than two orders of magnitude faster than the exact classifier with minor losses in quality.

V-soft margin multi-task learning logistic regression

Coordinate descent (CD) is an effective method for large scale classification problems with simple operations and fast convergence speed. In this paper, inspired by v-soft margin support vector machine and multi-task learning support vector machine for classification, a novel v-soft margin multi-task learning logistic regression (v-SMMTL-LR) for pattern classification is proposed to improve the generalization performance of logistic regression (LR). The dual of v-SMMTL-LR can be viewed as dual coordinate descent (CDdual) problem with equality constraint and then its large scale classification method named v-SMMTL-LR-CDdual is developed. The proposed method v-SMMTL-LR-CDdual can maximize the between-class margin and effectively improve the generalization performance of LR for large scale multi-task learning scenarios. Experimental results show that the proposed method v-SMMTL-LR-CDdual is effective for large scale multi-task datasets or comparatively high dimensional multi-task datasets and that it is competitive to other related single-task and multi-task learning algorithms.