Discriminant analysis and density estimation on the finite d-dimensional grid

Abstract

A new theoretical point of view is discussed here in the framework of density estimation. The discrete multivariate true density is viewed as a finite dimensional continuous random vector governed by a Markov random field structure. Estimating the density is then a problem of maximizing a conditional likelihood under a Bayesian framework. This maximization problem is expressed as a constrained optimization problem and is solved by an iterative fixed point algorithm. However, for time efficiency reasons, we have been interested in an approximate estimate f ̂ = Bπ of the true density f, where B is a stochastic matrix and π is the raw histogram. This estimate is obtained by developing f ̂ as a function of π around the uniform histogram π 0, using multivariate Taylor expansions for implicit functions ( f ̂ is actually an implicit function of π). The discrete setting of the problem allows us to get a simple analytical form for B. Although the approach is original, our density estimator is actually nothing else than a penalized maximum likelihood estimator. However, it appears to be more general than those proposed in the literature (Scott et al., 1980; Simonoff, 1983; Thompson and Tapia, 1990). In a second step, we investigate the discrimination problem on the same space, using the theory previously developed for density estimation. We also introduce an adaptive bandwidth depending on the k-nearest neighbours and we have chosen to optimize the leaving-one-out criterion. We have always kept in mind the practical implementation on a computer. Our final classification algorithm compares favourably in terms of error rate and time efficiency with other algorithms tested, including multinormal IMSL, nearest-neighbour, and convex hull classifiers. Comparisons were performed on satellite images.