Hierarchical discriminant regression

Abstract

The main motivation of this paper is to propose a classification and regression method for challenging high-dimensional data. The proposed technique casts classification problems and regression problems into a unified regression problem. This unified view enables classification problems to use numeric information in the output space that is available for regression problems but are traditionally not readily available for classification problems. A clustered subspace-based hierarchical discriminating regression (HDR) method is proposed. The major characteristics include: (1) Clustering is performed in both output space and input space at each internal node, termed doubly clustered. Clustering in the output space provides virtual labels for computing clusters in the input space. (2) Discriminants in the input space are automatically derived from the clusters in the input space. (3) A hierarchical probability distribution model is applied to the resulting discriminating subspace at each internal node. This realizes a coarse-to-fine approximation of probability distribution of the input samples, in the hierarchical discriminating subspaces. (4) To relax the per class sample requirement of traditional discriminant analysis techniques, a sample-size dependent negative-log-likelihood (NLL) is introduced. This new technique is designed for automatically dealing with small-sample applications, large-sample applications, and unbalanced-sample applications. (5) The execution of the HDR method is fast, due to the empirical logarithmic time complexity of the HDR algorithm. Although the method is applicable to any data, we report the experimental results for three types of data: synthetic data for examining the near-optimal performance, large raw face-image databases, and traditional databases with manually selected features along with a comparison with some major existing methods.