Mammographic density phenotypes, adjusted for age and body mass index (BMI), are strong predictors of breast cancer risk. BMI is associated with mammographic density measures, but the role of circulating sex hormone concentrations is less clear. We investigated the relationship between BMI, circulating sex hormone concentrations, and mammographic density phenotypes using Mendelian randomization (MR). We applied two-sample MR approaches to assess the association between genetically predicted circulating concentrations of sex hormones [estradiol, testosterone, sex hormone-binding globulin (SHBG)], BMI, and mammographic density phenotypes (dense and non-dense area). We created instrumental variables from large European ancestry-based genome-wide association studies and applied estimates to mammographic density phenotypes in up to 14,000 women of European ancestry. We performed analyses overall and by menopausal status. Genetically predicted BMI was positively associated with non-dense area (IVW: β = 1.79; 95% CI = 1.58, 2.00; p = 9.57 × 10-63) and inversely associated with dense area (IVW: β = - 0.37; 95% CI = - 0.51,- 0.23; p = 4.7 × 10-7). We observed weak evidence for an association of circulating sex hormone concentrations with mammographic density phenotypes, specifically inverse associations between genetically predicted testosterone concentration and dense area (β = - 0.22; 95% CI = - 0.38, - 0.053; p = 0.009) and between genetically predicted estradiol concentration and non-dense area (β = - 3.32; 95% CI = - 5.83, - 0.82; p = 0.009), although results were not consistent across a range of MR approaches. Our findings support a positive causal association between BMI and mammographic non-dense area and an inverse association between BMI and dense area. Evidence was weaker and inconsistent for a causal effect of circulating sex hormone concentrations on mammographic density phenotypes. Based on our findings, associations between circulating sex hormone concentrations and mammographic density phenotypes are weak at best.