Statistical mechanics of learning orthogonal signals for general covariance models

Abstract

Statistical mechanics techniques have proved to be useful tools in quantifying the accuracywith which signal vectors are extracted from experimental data. However, analysis haspreviously been limited to specific model forms for the population covarianceC, which may be inappropriate for real world data sets. In this paper we obtainnew statistical mechanical results for a general population covariance matrixC. For data setsconsisting of p sample points in we use the replica method to study the accuracy of orthogonal signal vectors estimatedfrom the sample data. In the asymptotic limit of at fixed α = p/N, we derive analytical results for the signal direction learning curves. In the asymptoticlimit the learning curves follow a single universal form, each displaying a retarded learningtransition. An explicit formula for the location of the retarded learning transition isobtained and we find marked variation in the location of the retarded learning transitiondependent on the distribution of population covariance eigenvalues. The results of thereplica analysis are confirmed against simulation.