The neurobiological differences between women who have experienced a peripartum episode and those who have only had episodes outside of this period are not well understood. 64 parous female patients with major depressive disorder that have either a positive (n=30) or negative (n=34) history of peripartum depression (PPD) underwent MRI acquisition to obtain structural brain images. An independent two-sample t-test comparing patients with and without a history of PPD was performed using voxel-based morphometry analysis (VBM). Additionally, polygenic risk scores (PRSs) for estradiol were calculated and a moderation analysis was conducted between 3 estradiol PRSs and PPD history status on extracted cluster volumes using IBM SPSS PROCESS macro. The VBM analysis identified larger grey matter volumes in bilateral clusters encompassing the putamen, pallidum, caudate, and thalamus in patients with PPD history compared to patients without a history. The moderation analysis identified a significant interaction of 2 estradiol PRSs and PPD history on grey matter cluster volumes with a positive effect in PPD women and a negative effect in women with no history of PPD. Our findings demonstrate that women who have experienced a peripartum episode are neurobiologically distinct from women who have no history of PPD in a cluster within the basal ganglia, an area important for motivation, decision-making, and emotional processing. Furthermore, we show that the genetic load for estradiol has a differing effect in this area based on PPD status which supports the claim that PPD is associated with sensitivity to sex steroid hormones.