Understanding the relationships between motor proficiency (MP) and physical fitness (PF) is important for the future health of children, although longitudinal findings of this nature are limited. This study explored the association between MP and PF during earlier childhood (6 and 9 years old) and later childhood (12 years old) in boys and girls. A stratified and randomized research design including a baseline and two follow-up time-point measures (2010, 2013, and 2016) were used. Primary school children (N = 374, boys = 178; girls = 196) were tested with the Bruininks-Oseretsky Test of Motor-Proficiency-2, Short Form, and the Test of Gross Motor Development- 2, at ages 6 and 9 years, and with the Progressive Aerobic Cardiovascular Endurance Run test at age 12. Spearman Rank Order Correlations and stepwise regression analyses were used to analyze the data. Although of a low magnitude, proof of a dynamic longitudinal, but a stable relationship between MP and PF were found and with different gender-specific relationships in this pathway of association. A small but significant percentage of variation in PF at 12 years could be explained by overall motor competence (MC) and object control skills (OCS) at the ages of 6 and 9 years. Among girls, the association with PF at 12 years was influenced by both OCS, and MC, although only MC contributed to the variation found in boys. Socio-economic status made an insignificant contribution at 6 years to later PF in boys, but not in girls. MC, including OCS during early and middle childhood can be considered as possible triggers of physical activity which again, can increase PF during later childhood. Obtaining early competence in these developmental areas is therefore important to promote positive and sustainable trajectories of health with long-term health outcomes.