Uniqueness theorems for kernel methods

Abstract

This paper has three goals: to find uniqueness theorems for some well-known kernel methods, and thereby to better understand their behavior; to derive theorems which apply to more general families of kernel methods; and to collect results which are hoped to be useful to workers who would like to prove uniqueness theorems for their own algorithms. Knowing when a solution is unique has the obvious advantage of removing the need for searching for better solutions, but it can also provide useful insight into the algorithms themselves. We point out that for any convex optimization problem, if the objective function can be written as the sum of a function which is strictly convex in some subset S of the variables, and a second, convex function, then the solution is unique in the S variables. We give necessary and sufficient conditions for the solutions of the SVM and ν-SVM classification and regression problems to be unique. We show how the analysis can be extended to a large class of kernel methods with a convex objective function and linear constraints. For SVMs, we show that if the solution is not unique, all support vectors are necessarily at bound, and the margin width is fixed for the set of solutions. The case of ν-SVMs is different, in that the margin width and offset can both change within a set of non-unique solutions, and that in the case of ν-SVM classifiers, the solution can be unstable with respect to small changes in ν; we show that this kind of instability, and the occurrence of non-unique solutions, are closely linked. We give simple examples illustrating these findings. Finally, the reasoning also leads to a simple method for computing the offset for an SVM classifier or regression model when the usual method for determining the offset does not apply.