SIGEST

Abstract

This issue's SIGEST paper, “The Trace Ratio Optimization Problem,” by T. T. Ngo, M. Bellalij, and Y. Saad, is from the SIAM Journal on Matrix Analysis and Applications. The mathematical problem is to maximize a nonlinear function over a matrix manifold. The difficult formulation, tackled in this paper, has as its data two $p \times p$ positive definite matrices $A$ and $B$. The problem is to maximize the quotient of \[ \fracTr(V^T A V)Tr(V^T B V) \] over the manifold of $n \times p$ matrices with orthonormal columns. A different and simpler formulation adds the constraint $\mbox{Tr}(V^T B V) = 1$. Many previous papers considered this form more tractable. However, the results are not as useful as those from the quotient formulation. In this paper the authors show how to maximize the quotient more efficiently than was possible even for the simplified formulation. The (very elegant) algorithm's two important parts are a Lanczos process to find the $p$ largest eigenvalues of an $n \times n$ matrix and a Newton iteration for a scalar nonlinear equation. The application is to supervised learning. A classic example is Fischer linear discriminant analysis, where one seeks to identify a hyperplane that separates two or more datasets. The solution $V$ is a projection of the data onto an “optimal” lower dimensional space of dimension $p$. The matrices $A$ and $B$ are built from the data. The numerator in the fraction represents the quality of separation of the classes, and the denominator how well each class is clustered in the projected space. These authors clearly quantify these concepts in the introduction, which is a generous expansion of the introduction of the original paper. We teach the Lanczos process and Newton's method in introductory courses. These two ideas are the basis for the results in this paper. This issue's SIGEST paper is a compelling example of how methods from elementary courses can be applied to very nontrivial problems and produce surprising results.