Prior studies have reported long-term differences in brain structure (brain morphometry) as being associated with cumulative concussion and contact sport participation. There is emerging evidence to suggest that similar effects of prior concussion and contact sport participation on brain morphometry may be present in younger cohorts of active athletes. We investigated the relationship between prior concussion and primary sport participation with subcortical and cortical structures in active collegiate contact sport and non-contact sport athletes. Contact sport athletes (CS; N = 190) and matched non-contact sport athletes (NCS; N = 95) completed baseline clinical testing and participated in up to four serial neuroimaging sessions across a 6-months period. Subcortical and cortical structural metrics were derived using FreeSurfer. Linear mixed-effects (LME) models examined the effects of years of primary sport participation and prior concussion (0, 1+) on brain structure and baseline clinical variables. Athletes with prior concussion across both groups reported significantly more baseline concussion and psychological symptoms (all ps < 0.05). The relationship between years of primary sport participation and thalamic volume differed between CS and NCS (p = 0.015), driven by a significant inverse association between primary years of participation and thalamic volume in CS (p = 0.007). Additional analyses limited to CS alone showed that the relationship between years of primary sport participation and dorsal striatal volume was moderated by concussion history (p = 0.042). Finally, CS with prior concussion had larger hippocampal volumes than CS without prior concussion (p = 0.015). Years of contact sport exposure and prior concussion(s) are associated with differences in subcortical volumes in young-adult, active collegiate athletes, consistent with prior literature in retired, primarily symptomatic contact sport athletes. Longitudinal follow-up studies in these athletes are needed to determine clinical significance of current findings.