Sedentary lifestyle morbidities are common among children with congenital heart disease (CHD). Understanding the physical activity trajectory from early childhood could enhance timing and effectiveness of interventions. We recruited 154 children (56% male) at 12 to 47 months of age for this prospective, longitudinal, observational study. Physical activity and sedentary behaviour (7-day accelerometry) and motor skill (Peabody Developmental Motor Scales-2) were assessed every 8 months until 5 years of age and then annually. Mixed-effect repeated measures regression models described outcome trajectories across study assessments. Children had innocent heart murmurs (n= 28), CHD with insignificant hemodynamics not requiring treatment (n= 47), CHD treated by catheterization or surgery without cardiopulmonary bypass (n= 31), or CHD treated surgically with bypass (n= 48). Motor skill was age appropriate (Peabody 49.0 ± 8.4), but participants had lower physical activity (143 ± 41 minutes per day) and higher sedentary time (598 ± 89 minutes per day) than healthy peers, starting at 18 months of age. Movement behaviours were not related to treatment group (P > 0.10), and physical activity was below the recommended 180 minutes per day. Over time, physical activity, sedentary time, and motor skills were primarily related to the baseline measure of each outcome (P < 0.001). Children with simple or complex CHD or innocent heart murmurs have increased risk for sedentary lifestyles. Their physical activity and sedentary behaviours are established before 2 years of age, persist until school age, and are unrelated to motor skills. These results emphasize the need for interventions targeting the youngest children seen in cardiac clinics, regardless of diagnoses of CHD or innocent murmur.