In this paper we consider pseudo-Riemannian spaces of arbitrary signature for which all of the polynomial curvature invariants vanish (VSI spaces). Using an algebraic classification of pseudo-Riemannian spaces in terms of the boost-weight decomposition, we first show more generally that a space which is not characterized by its invariants must possess the SG1-property. As a corollary, we then show that a VSI space must possess the NG-property (these results are the analogues of the alignment theorem, including corollaries, for Lorentzian spacetimes). As an application we classify all 4D neutral VSI spaces and show that these belong to one of two classes: (1) those that possess a geodesic, expansion-free, shear-free, and twist-free null congruence (Kundt metrics), or (2) those that possess an invariant null plane (Walker metrics). By explicit construction we show that the latter class contains a set of VSI metrics which have not previously been considered in the literature.