The purpose of this study was to characterize sex differences in the relationship between body composition and cardiac structure and function. In secondary analyses, we explored pathophysiologic mediators of these relationships. In a cross-sectional analysis of 25,063 UK Biobank participants (54% female, median age 55 years), the sex-specific associations of visceral adipose tissue volume (VAT), appendicular lean mass (ALM), and muscle fat infiltration (MFI) with cardiac magnetic resonance (CMR) measures of cardiac structure and function were assessed using linear regression models. Using causal mediation analysis, 10 biomarkers were explored as mediators of the relationship between adipose depots and cardiac parameters. VAT was associated with increased left ventricular mass (LVM; βwomen = 0.54, βmen = 0.00, pint = 0.01) and wall thickness (βwomen = 0.12, βmen = 0.08, pint<0.001) in women only. A similar sex-specific pattern was observed for MFI effects on LVM (βwomen = 0.44, βmen = 0.03, pint<0.001). ALM was associated with increased LVM and LV volumes in both women and men. In mediation analyses, insulin resistance as measured by triglycerides/high density lipoprotein ratio was a potential partial mediator of VAT effects on chamber dimensions. In the largest and most rigorous analyses of body composition and cardiac parameters to date, we demonstrated that VAT is associated with increased LVM and wall thickness in women but not in men. MFI association with cardiac parameters was similar to VAT, significant in women but not in men.