Kernel Minimum Squared Error Research Articles

Sparse kernel minimum squared error using Householder transformation and givens rotation

Two obvious limitations exist for baseline kernel minimum squared error (KMSE): lack of sparseness of the solution and the ill-posed problem. Previous sparse methods for KMSE have overcome the second limitation using a regularization strategy, which introduces an increase in the computational cost to determine the regularization parameter. Hence, in this paper, a constructive sparse algorithm for KMSE (CS-KMSE) and its improved version (ICS-KMSE) are proposed which will simultaneously address the two limitations described above. CS-KMSE chooses the training samples that incur the largest reductions on the objective function as the significant nodes on the basis of the Householder transformation. In contrast with CS-KMSE, there is an additional replacement mechanism using Givens rotation in ICS-KMSE, which results in ICS-KMSE giving better performance than CS-KMSE in terms of sparseness. CS-KMSE and ICS-KMSE do not require the regularization parameter at all before they begin to choose significant nodes, which is beneficial since it saves on the model selection time. More importantly, CS-KMSE and ICS-KMSE terminate their procedures with an early stopping strategy that acts as an implicit regularization term, which avoids overfitting and curbs the sparse level on the solution of the baseline KMSE. Finally, in comparison with other algorithms, both ICS-KMSE and CS-KMSE have superior sparseness, and extensive comparisons confirm their effectiveness and feasibility.

Generalization improvement for regularized least squares classification

In the past decades, regularized least squares classification (RLSC) is a commonly used supervised classification method in the machine learning filed because it can be easily resolved through the simple matrix analysis and achieve a close-form solution. Recently, some studies conjecture that the margin distribution is more crucial to the generalization performance. Moreover, from the view of margin distribution, RLSC only considers the first-order statistics (i.e., margin mean) and does not consider the actual higher-order statistics of margin distribution. In this paper, we propose a novel RLSC which takes into account the actual second-order (i.e., variance) information of margin distribution. It is intuitively expected that small margin variance will improve the generalization performance of RLSC from a geometric view. We incorporate the margin variance into the objective function of RLSC and achieve the optimal classifier by minimizing the margin variance. To evaluate the performance of our algorithm, we conduct a series of experiments on several benchmark datasets in comparison with RLSC, kernel minimum squared error, support vector machine and large margin distribution machine. And the empirical results verify the effectiveness of our algorithm and indicate that the margin distribution is helpful to improve the classification performance.

Enhanced manifold regularization for semi-supervised classification.

Manifold regularization (MR) has become one of the most widely used approaches in the semi-supervised learning field. It has shown superiority by exploiting the local manifold structure of both labeled and unlabeled data. The manifold structure is modeled by constructing a Laplacian graph and then incorporated in learning through a smoothness regularization term. Hence the labels of labeled and unlabeled data vary smoothly along the geodesics on the manifold. However, MR has ignored the discriminative ability of the labeled and unlabeled data. To address the problem, we propose an enhanced MR framework for semi-supervised classification in which the local discriminative information of the labeled and unlabeled data is explicitly exploited. To make full use of labeled data, we firstly employ a semi-supervised clustering method to discover the underlying data space structure of the whole dataset. Then we construct a local discrimination graph to model the discriminative information of labeled and unlabeled data according to the discovered intrinsic structure. Therefore, the data points that may be from different clusters, though similar on the manifold, are enforced far away from each other. Finally, the discrimination graph is incorporated into the MR framework. In particular, we utilize semi-supervised fuzzy c-means and Laplacian regularized Kernel minimum squared error for semi-supervised clustering and classification, respectively. Experimental results on several benchmark datasets and face recognition demonstrate the effectiveness of our proposed method.

A method of combining forward with backward greedy algorithms for sparse approximation to KMSE

The prefitting and backfitting methods are commonly used to sparsify the full solution of naive kernel minimum squared error. As known to us, the forward learning methods including prefitting and backfitting only assist us in finding the suboptimal solutions. To enhance the testing real time further, in this paper by virtue of the idea of incorporating the backward learning algorithm into the forward learning algorithm, two improved schemes on the basis of prefitting and backfitting are proposed. Compared with the original versions, two improved algorithms obtain fewer significant nodes, which indicates much better testing real time. Due to the addition of the backward learning to the forward learning, the proposed algorithms need more training computational costs. Investigations on benchmark data sets and a robot arm example are reported to demonstrate the improved effectiveness.

Improved the minimum squared error algorithm for face recognition by integrating original face images and the mirror images

In order to improve the accuracy of face recognition and solve the problem of various poses and illuminations, we proposed an improved minimum squared error (IMSE) classification algorithm. Firstly, the mirror faces of the original training faces are generated through row preserved and columns flipped in the left/right direction. Secondly, the minimum squared error (MSE) algorithm is performed on both original faces and the mirror faces. Thirdly, the predicted errors of the test sample and standard class labels are obtained. In addition, the residual between the predicted labels of the test sample and each training sample can also be calculated. At last, the correct class can be determined by fusing the predicted errors and residuals. We also promoted the IMSE algorithm to the kernel MSE algorithm and proposed an improved kernel minimum squared error (IKMSE) algorithm for face recognition. The experimental results show our proposed IMSE and IKMSE algorithm are more robust than the conventional MSE and KMSE algorithm, respectively. In addition, our proposed algorithms improve the accuracy of face recognition effectively.

Multi-class semi-supervised kernel minimum squared error for face recognition

Kernel Minimum Squared Error (KMSE) has become a hot topic in machine learning and pattern recognition in the past years. However, KMSE is essentially a binary classifier and one-against-all and one-against-one strategies are usually employed to deal with multi-class problems. In this situation, KMSE needs to resolve multiple equations with the high computation complexity. Meanwhile, labeled examples are usually insufficient and unlabeled ones are abundant in many real-world applications. Therefore in this paper, we introduce a novel multi-class semi-supervised KMSE algorithm, called multi-class Laplacian regularized KMSE (McLapKMSE). Compared to KMSE and semi-supervised KMSE, we need resolve one equation at once and therefore the method has the lower complexity. The experiments on face recognition are conducted to illustrate that our algorithm can achieve the comparable performance and lower complexity in contrast to the other supervised and semi-supervised methods.

A risk degree-based safe semi-supervised learning algorithm

Semi-supervised learning has attracted much attention in machine learning field over the past decades and a number of algorithms are proposed to improve the performance by exploiting unlabeled data. However, unlabeled data may hurt performance of semi-supervised learning in some cases. It is instinctively expected to design a reasonable strategy to safety exploit unlabeled data. To address the problem, we introduce a safe semi-supervised learning by analyzing the different characteristics of unlabeled data in supervised and semi-supervised learning. Our intuition is that unlabeled data may be often risky in semi-supervised setting and the risk degree are different. Hence, we assign different risk degree to unlabeled data and the risk degree serve as a sieve to determine the exploiting way of unlabeled data. The unlabeled data with high risk should be exploited by supervised learning and the other should be used for semi-supervised learning. In particular, we utilize kernel minimum squared error (KMSE) and Laplacian regularized KMSE for supervised and semi-supervised learning, respectively. Experimental results on several benchmark datasets illustrate the performance of our algorithm is never inferior to that of KMSE and indicate the effectiveness and efficiency of our algorithm.

Towards a probabilistic semi-supervised Kernel Minimum Squared Error algorithm

Recently, semi-supervised learning has received much attention in data mining and machine learning, and a number of algorithms are proposed to discuss how to make good use of the unlabeled data. Some algorithms deal with the unlabeled data in an exact way, in which each unlabeled sample is assigned to one single class and then treated as a labeled sample. Other algorithms use the unlabeled data to regularize the objective function but do not explicitly model the influence of the unlabeled data towards different classes. In many applications, however, the unlabeled data may be ambiguous and belong to multiple classes with different probabilities. Based on this assumption, this paper presents a Probabilistic Laplacian-regularized Kernel Minimum Squared Error algorithm (named PrLapKMSE), in which the probabilities of the unlabeled data belonging to different classes are adaptively generated. “Adaptively” means that these probabilities are recalculated iteratively along with the reformulated objective function so that the unlabeled data may have increasingly accurate effects on the semi-supervised learning procedure. Experimental results on several simulated and real-world datasets illustrate the effectiveness of our algorithm.

Feature Extraction for Kernel Minimum Squared Error by Sparsity Shrinkage

In this paper, a sparsity based model is proposed for feature selection in kernel minimum squared error (KMSE). By imposing a sparsity shrinkage term, we formulate the procedure of subset selection as an optimization problem. With the chosen small portion of training examples, the computational burden of feature extraction is largely alleviated. Experimental results conducted on several benchmark datasets indicate the effectivity and efficiency of our method.

Laplacian regularized kernel minimum squared error and its application to face recognition

Kernel minimum squared error (KMSE) has been receiving much attention in data mining and pattern recognition in recent years. Generally speaking, training a KMSE classifier, which is a kind of supervised learning, needs sufficient labeled examples. However, labeled examples are usually insufficient and unlabeled examples are abundant in real-world applications. In this paper, we introduce a semi-supervised KMSE algorithm, called Laplacian regularized KMSE (LapKMSE), which explicitly exploits the manifold structure. We construct a p nearest neighbor graph to model the manifold structure of labeled and unlabeled examples. Then, LapKMSE incorporates the structure information of labeled and unlabeled examples in the objective function of KMSE by adding a Laplacian regularization term. As a result, the labels of labeled and unlabeled examples vary smoothly along the geodesics on the manifold. Experimental results on several synthetic and real-world datasets illustrate the effectiveness of our algorithm. Finally our algorithm is applied to face recognition and achieves the comparable results compared to the other supervised and semi-supervised methods.

Incremental kernel minimum squared error (KMSE)

Kernel minimum squared error (KMSE) is a simple and efficient learning algorithm extensively used by the machine learning community. The solution of KMSE is typically not sparse, which may be harmful for real time applications. Current proposed algorithms show hindrances when dealing with sample flow learning systems. A significant improvement utilizes incremental learning, namely IKMSE, to achieve better sparsity however it is not enough. We propose two additional adaptive incremental learning algorithms, viz. CRKMSE and PRKMSE. CRKMSE adopts a complete reduction strategy while PRKMSE uses the partial reduction. Generally, PRKMSE is superior to CRKMSE in terms of training time. And PRKMSE owns a better real time than CRKMSE in the testing phase due to fewer significant nodes. Finally, a lot of experiments are reported to confirm our viewpoints in this paper.

Semiparametric spatial effects kernel minimum squared error model for predicting housing sales prices

Semiparametric regression models have been extensively used to predict housing sales prices, but semiparametric kernel machines with spatial effect have not been studied yet. This paper proposes the semiparametric spatial effect kernel minimum squared error model (SSEKMSEM) and the semiparametric spatial effect least squares support vector machine (SSELS-SVM) for estimating a hedonic price function and compares the price prediction performance with the conventional parametric models and a semiparametric generalized additive model (GAM). This paper utilizes two data sets. One is a large data set representing 5966 single-family residential home sales between July 2000 and August 2008 from Pitt County, North Carolina. The other is a data set of residential property sales records from September 2000 to September 2004 in Carteret County, North Carolina. The results show that the SSEKMSEM and SSELS-SVM outperform the parametric counterparts and the semiparametric GAM in both in-sample and out-of-sample price predictions, indicating that these kernel machines can be useful for measurement and prediction of housing sales prices.

Improvement of the kernel minimum squared error model for fast feature extraction

The kernel minimum squared error (KMSE) expresses the feature extractor as a linear combination of all the training samples in the high-dimensional kernel space. To extract a feature from a sample, KMSE should calculate as many kernel functions as the training samples. Thus, the computational efficiency of the KMSE-based feature extraction procedure is inversely proportional to the size of the training sample set. In this paper, we propose an efficient kernel minimum squared error (EKMSE) model for two-class classification. The proposed EKMSE expresses each feature extractor as a linear combination of nodes, which are a small portion of the training samples. To extract a feature from a sample, EKMSE only needs to calculate as many kernel functions as the nodes. As the nodes are commonly much fewer than the training samples, EKMSE is much faster than KMSE in feature extraction. The EKMSE can achieve the same training accuracy as the standard KMSE. Also, EKMSE avoids the overfitting problem. We implement the EKMSE model using two algorithms. Experimental results show the feasibility of the EKMSE model.

Pruning least objective contribution in KMSE

Although kernel minimum squared error (KMSE) is computationally simple, i.e., it only needs solving a linear equation set, it suffers from the drawback that in the testing phase the computational efficiency decreases seriously as the training samples increase. The underlying reason is that the solution of Naïve KMSE is represented by all the training samples in the feature space. Hence, in this paper, a method of selecting significant nodes for KMSE is proposed. During each calculation round, the presented algorithm prunes the training sample making least contribution to the objective function, hence called as PLOC-KMSE. To accelerate the training procedure, a batch of so-called nonsignificant nodes is pruned instead of one by one in PLOC-KMSE, and this speedup algorithm is named MPLOC-KMSE for short. To show the efficacy and feasibility of the proposed PLOC-KMSE and MPLOC-KMSE, the experiments on benchmark data sets and real-world instances are reported. The experimental results demonstrate that PLOC-KMSE and MPLOC-KMSE require the fewest significant nodes compared with other algorithms. That is to say, their computational efficiency in the testing phase is best, thus suitable for environments having a strict demand of computational efficiency. In addition, from the performed experiments, it is easily known that the proposed MPLOC-KMSE accelerates the training procedure without sacrificing the computational efficiency of testing phase to reach the almost same generalization performance. Finally, although PLOC and MPLOC are proposed in regression domain, they can be easily extended to classification problem and other algorithms such as kernel ridge regression.

A fast method of feature extraction for kernel MSE

In this paper, a fast method of selecting features for kernel minimum squared error (KMSE) is proposed to mitigate the computational burden in the case where the size of the training patterns is large. Compared with other existent algorithms of selecting features for KMSE, this iterative KMSE, viz. IKMSE, shows better property of enhancing the computational efficiency without sacrificing the generalization performance. Experimental reports on the benchmark data sets, nonlinear autoregressive model and real problem address the efficacy and feasibility of the proposed IKMSE. In addition, IKMSE can be easily extended to classification fields.