Although behavioral studies show large improvements in arithmetic skills in elementary school, we do not know how brain structure supports math gains in typically developing children. While some correlational studies have investigated the concurrent association between math performance and brain structure, such as gray matter volume (GMV), longitudinal studies are needed to infer if there is a causal relation. Although discrepancies in the literature on the relation between GMV and math performance have been attributed to the different demands on quantity vs. retrieval mechanisms, no study has experimentally tested this assumption. We defined regions of interests (ROIs) associated with quantity representations in the bilateral intraparietal sulcus (IPS) and associated with the storage of arithmetic facts in long-term memory in the left middle and superior temporal gyri (MTG/STG), and studied associations between GMV in these ROIs and children's performance on operations having greater demands on quantity vs. retrieval mechanisms, namely subtraction vs. multiplication. The aims of this study were threefold: First, to study concurrent associations between GMV and math performance, second, to investigate the role of GMV at the first time-point (T1) in predicting longitudinal gains in math skill to the second time-point (T2), and third, to study whether changes in GMV over time were associated with gains in math skill. Results showed no concurrent association between GMV in IPS and math performance, but a concurrent association between GMV in left MTG/STG and multiplication skill at T1. This association showed that the higher the GMV in this ROI, the higher the children's multiplication skill. Results also revealed that GMV in left IPS and left MTG/STG predicted longitudinal gains in subtraction skill only for younger children (approximately 10 years old). Whereas higher levels of GMV in left IPS at T1 predicted larger subtraction gains, higher levels of GMV in left MTG/STG predicted smaller gains. GMV in left MTG/STG did not predict longitudinal gains in multiplication skill. No significant association was found between changes in GMV over time and longitudinal gains in math. Our findings support the early importance of brain structure in the IPS for mathematical skills that rely on quantity mechanisms.