Information about the domains of physical activity (PA) that are most prone to decline between late childhood (11y), early adolescence (13y), and mid-adolescence (15y) may support more targeted health promotion strategies. This study explored longitudinal trends in nonorganized PA, organized PA, active transport and active chores/work between childhood and adolescence, and potential sociodemographic moderators of changes. Data were sourced from the Longitudinal Study of Australian Children (n = 4108). Participation in PA domains was extracted from youth time-use diaries. Potential moderators were sex, Indigenous status, language spoken at home, socioeconomic position, and geographical remoteness. A large quadratic decline in nonorganized PA (-48min/d, P < .001) was moderated by sex (β = 5.55, P = .047) and home language (β = 8.55, P = .047), with girls (-39min/d) and those from a non-English speaking background (-46min/d) declining more between 11 and 13 years. Active chores/work increased between 11 and 13 years (+4min/d, P < .001) and then stabilized. Active transport increased among boys between 11 and 13 years (+6min/d, P < .001) and then declined between 13 and 15 years (-4min/d, P < .001). Organized PA remained stable. The longitudinal decline in PA participation may be lessened by targeting nonorganized PA between childhood and adolescence. Future interventions may target girls or those from non-English speaking backgrounds during this transition.