By the end of primary school, children are expected to acquire a range of mathematical skills that progressively develop. This study aimed to gain insight into how a large number of numerical and geometrical measures are grouped and whether the structures shift or remain invariant along child’s development based on the data obtained from a sample of 1346 s to sixth grade children. On the basis of correlation analyses and exploratory factor analyses, we came up with an invariant four-factor structure for each grade. The four factors obtained were the following: (a) early and retrieval skills (subitizing, enumeration, number facts) (b) transcoding and ordinality skills, (c) numerical advanced skills and (d) visual-spatial advanced skills. Confirmatory factor analyses showed that the four-factor model fits well to the data (RMSEA < 0.06, CFI > 0.95, GFI > 0.93, SRMR < 0.05) at all grades. The associations between the mathematical measures were captured in each factor and the revealed invariant structure is discussed and compared with previous classification models of mathematical skills. Given that the difficulties in learning mathematics is currently being viewed as a continuum of academic abilities instead of a distinct problem, the underlying invariant four-factor structure can facilitate experts and educators to better understand how a broad area of mathematical skills are related across the primary education, in order to carry our comprehensive assessment of both the mathematical strengths and weaknesses of their students and to apply the appropriate customized teaching strategies.