The effect of a vegetarian diet on bone health remains controversial. This retrospective medical record review compared changes in bone mineral density (BMD) and trabecular bone score (TBS) between vegetarian and non-vegetarian middle-aged and older women who underwent two general health examinations (T1 and T2) that were approximately three years apart. Generalized estimating equations were used to compare the change in lumbar spine and bilateral hip BMD and TBS over time. At T1, the mean age of the patients was 56.6 years (standard deviation 9.7 years) and the mean interval between T1 and T2 was 2.7 years. For women aged 40–55 years, compared with non-vegetarians, vegetarians were significantly associated with a larger reduction in lumbar spine BMD (p < 0.001) and left hip femoral neck BMD (p = 0.015) over the three-year interval. On the contrary, changes in BMD were not significant at any site in women aged ≥ 56 years. Moreover, the changes in BMD and TBS over the three-year interval did not significantly differ between vegetarian and non-vegetarian women aged 65–90 years. In conclusion, for women aged 40–55 years, vegetarian diets reduced bone quantity, as measured by BMD, but not bone quality, as measured by TBS.