A comprehensive analysis of associations between physical fitness and brain structure in young adulthood is lacking, and further, it is unclear the degree to which associations between physical fitness and brain health can be attributed to a common genetic pathway or to environmental factors that jointly influences physical fitness and brain health. This study examined genotype-confirmed monozygotic and dizygotic twins, along with non-twin full-siblings to estimate the contribution of genetic and environmental factors to variation within, and covariation between, physical fitness and brain structure. Participants were 1,065 young adults between the ages of 22 and 36 from open-access Young Adult Human Connectome Project (YA-HCP). Physical fitness was assessed by submaximal endurance (2-min walk test), grip strength, and body mass index. Brain structure was assessed using magnetic resonance imaging on a Siemens 3T customized ‘Connectome Skyra’ at Washington University in St. Louis, using a 32-channel Siemens head coil. Acquired T1-weighted images provided measures of cortical surface area and thickness, and subcortical volume following processing by the YA-HCP structural FreeSurfer pipeline. Diffusion weighted imaging was acquired to assess white matter tract integrity, as measured by fractional anisotropy, following processing by the YA-HCP diffusion pipeline and tensor fit. Following correction for multiple testing, body mass index was negatively associated with fractional anisotropy in various white matter regions of interest (all | z| statistics > 3.9) and positively associated with cortical thickness within the right superior parietal lobe (z statistic = 4.6). Performance-based measures of fitness (i.e., endurance and grip strength) were not associated with any structural neuroimaging markers. Behavioral genetic analysis suggested that heritability of white matter integrity varied by region, but consistently explained >50% of the phenotypic variation. Heritability of right superior parietal thickness was large (∼75% variation). Heritability of body mass index was also fairly large (∼60% variation). Generally, to of the correlation between brain structure and body mass index could be attributed to heritability effects. Overall, this study suggests that greater body mass index is associated with lower white matter integrity, which may be due to common genetic effects that impact body composition and white matter integrity.