There is a gap in knowledge regarding the polygenic underpinnings of brain anomalies observed in youth bipolar disorder (BD). This study examined the association of a polygenic risk score for BD (BD-PRS) with grey matter structure and white matter integrity in youth with and without BD. 113 participants were included in the analyses, including 78 participants with both T1-weighted and diffusion-weighted MRI images, 32 participants with T1-weighted images only, and 3 participants with diffusion-weighted images only. BD-PRS was calculated using PRS-CS-auto and was based on independent adult genome-wide summary statistics. Vertex- and voxel-wise analyses examined the associations of BD-PRS with grey matter metrics (cortical volume [CV], cortical surface area [CSA], cortical thickness [CTh]) and fractional anisotropy [FA] in the combined sample, and separately in BD and HC. In the combined sample of participants with T1-weighted images (n = 110, 66 BD, 44 HC), higher BD-PRS was associated with smaller grey matter metrics in frontal and temporal regions. In within-group analyses, higher BD-PRS was associated with lower CTh of frontal, temporal, and fusiform gyrus in BD, and with lower CV and CSA of superior frontal gyrus in HC. In the combined sample of participants with diffusion-weighted images (n = 81, 49 BD, 32 HC), higher BD-PRS was associated with lower FA in widespread white matter regions. In summary, BD-PRS calculated based on adult genetic data was negatively associated with grey matter structure and FA in youth in regions implicated in BD, which may suggest neuroimaging markers of vulnerability to BD. Future longitudinal studies are needed to examine whether BD-PRS predicts neurodevelopmental changes in BD vs. HC and its interaction with course of illness and long-term medication use.