A broad variety of domain-specific (e.g., non-symbolic magnitude comparison and arithmetic) and domain-general (e.g., spatial skills and inhibition) skills have been identified as precursors to mathematics achievement. However, due to the increasing complexity of mathematics education with age and the associated increase in cognitive demands, it can be assumed that a divergent set of skills is predictive of mathematics achievement at different ages. This cross-sectional study in children aged 3, 5, and 7 years aims at identifying the differential contribution of domain-specific and domain-general contributors to mathematics and delineating their developmental dynamics. Our results reveal a consistent role for non-symbolic magnitude comparison across all age groups, non-symbolic arithmetic starting from the age of 5 years and visuospatial memory only in 5-year-olds. These findings support the notion that mathematics cannot be conceived as a unitary cognitive skill and provide a fine-grained analysis of the cognitive requirements of mathematical skills at different ages. On a more general note, they are in line with the idea that the ANS provides a critical scaffold for the development of mathematical skills but challenge the view that all ANS measures tap into the same underlying process.