This study aimed to explore the association between bone mineral density and physical frailty including nutrition, muscle mass, and oral function. We included participants aged 35–80 years and examined their bone mineral density, serum albumin level, body composition, and variance of hue (VOH) of two-colored gum. We also used the geriatric oral health assessment index (GOHAI). These data were used to calculate the geriatric nutritional risk index (GNRI) and skeletal muscle index (SMI). Multinomial logistic regression was performed to assess the relationship between GNRI, SMI, VOH, GOHAI, and bone mineral density after adjusting for comorbidities, including hypertension, diabetes mellitus, and previous bone fracture. We included 228 participants and classified their bone mineral density as normal, osteopenic, or osteoporotic. Older age (odds ratio (OR) 1.15, 95% confidence interval (CI) [1.08, 1.23]), low GNRI (OR 0.90, 95% CI [0.83, 0.98]), low SMI (OR 0.43, 95% CI [0.27, 0.68]), and high VOH (OR 1.08, 95% CI [1.01, 1.17]) were significantly associated with osteoporosis. Older age (OR 1.08, 95% CI [1.04, 1.11]) and low GNRI (OR 0.93, 95% CI [0.87, 0.99]) were significantly associated with osteopenia. GNRI, SMI, and VOH were significantly associated with osteoporosis among male participants. Although the multinomial logistic regression analysis indicated that GNRI, SMI, VOH, and GOHAI were not significantly associated with osteoporosis or osteopenia among female participants, the demographic distribution showed that older age, low GNRI, and low SMI were significantly associated with bone mineral density decline. Physical frailty, including nutritional decline, muscle mass loss, and poor oral status, is associated with low bone density. This easy-to-use tool can be used to detect osteoporosis early and to prevent osteoporosis and osteoporosis-related fractures.