ON ASYMPTOTICALLY OPTIMAL ALGORITHMS FOR PATTERN CLASSIFICATION PROBLEMS

Abstract

In recent years considerable interests have been given to the pattern classification problem. This problem includes three main aspects, the engineering aspect, the artificial intelligence aspect and the analytical aspect (c. f. [4]). The analytical one is concerned with the mathematical techniques of decision, estimation and optimization under the uncertainty of information. In this paper, our interest will be concentrated on the analytical aspect, especially on the estimation of a discriminant function which is optimum in the sense of the Bayes rule, minize the probability of misclassification, but which is unknown to us. We shall call such the discriminant function an optimal discriminant function (o. d. 1.). In previous works (e. g. [4], [7], [8], [9] and [11]), under the given situation of a sequence of observed patterns corectly classified by an external indicator, authors tried to obtain algorithms for finding the o. d. f. on the basis of the training sequence. In general, this approach has been called learning with a teacher in the pattern classification problem. And the method of approach which we shall appeal to in this paper is this case. The problem which will be treated in this paper is to classify the observed patterns into two categories, and the procedure which will be used is an application of the stochastic approximation method introduced by H. Robbins and S. Monro. Our object is to construct, on the basis of the given training sequence, estimates of the o. d. f. which are asymptotically optimal in the sense that the probabilities of misclassification from the estimates converge (with probability one or in the mean) to the probability of misclassification from the o. d. f. if it is known to us. This paper consists of five sections. In Section 2, we shall give definitions of the optimal discriminant function and of asymptotically optimal estimates to the o. d. f., and we shall prepare several lemmas to be used throughout subsequent sections. In Section 3, we shall treat the case when the o. d. f. is assumed to belong in the L2 space, and give an algorithm for constructing the asymptotically optimal estimates. J. V. Ryzin [7] treated this case, but his algorithm was not recursive and his convergence of the asymptotically optimal estimates was the one in the