Assessment of the effective rank of a (co)variance matrix: A non‐parametric goodness‐of‐fit test

Abstract

AbstractEach eigenvector of the dispersion matrix [X]T [X] was shown to be a partial predictor of the original data matrix [X], the sum of the predictions from the individual principal components being equal to the expectance of [X]. By comparing the distributions of the members of two neighbouring predicted matrices, [X̃]1…i and [X̃]1…i+1 (i.e. the sums of the first i and i + 1 individual predictions respectively), it was shown that they should be indistinguishable provided that i is equal to or greater than the effective rank of [X], and significantly different otherwise. This was confirmed by analysing the visible absorption spectra of methyl orange and methyl red solutions as well as the Raman spectra of Na2SO4 and MgSO4 solutions. On the grounds of these findings, a non‐parametric goodness‐of‐fit test for assessing the effective rank of [X] was proposed which proved to be comparatively conservative and more robust than most currently used tests.