Previous research has found an association of low bone mineral density (BMD) and regional gray matter (GM) volume loss in Alzheimer's disease (AD). We were interested whether BMD is associated with GM volume decrease in brains of a healthy elderly population from the UK Biobank. T1-weighted images from 5,518 women (MAge = 70.20, SD = 3.54; age range: 65-82 years) and 7,595 men (MAge = 70.84, SD = 3.68; age range: 65-82 years) without neurological or psychiatric impairments were included in voxel-based morphometry (VBM) analysis in CAT12 with threshold-free-cluster-enhancement (TFCE) across the whole brain. We found a significant decrease of GM volume in women in the superior frontal gyri, middle temporal gyri, fusiform gyri, temporal poles, cingulate gyri, precunei, right parahippocampal gyrus and right hippocampus, right ventral diencephalon, and right pre- and postcentral gyrus. Only small effects were found in men in subcallosal area, left basal forebrain and entorhinal area. BMD is associated with low GM volume in women but less in men in regions afflicted in the early-stages of AD even in a sample without neurodegenerative diseases.