The relationship between bone mass and muscle mass may be due to the site-specific effects of loading on bone in adults and to lifestyle, nutritional, and hormonal factors. Another hypothesis is that the maintenance with aging of both appendicular muscle and bone mass may be determined by factors independent of all these previous factors, including genetic factors. In 160 healthy men aged 20 to 72 years, we recorded femoral neck bone mineral density (FN BMD), relative appendicular skeletal muscle mass [RASM; appendicular skeletal muscle mass (kg) / height (cm)], age, body mass, maximum grip and knee extension strength, lifetime physical activities, calcium intake, tobacco smoking, and serum parathyroid hormone (PTH), estradiol (E2), free testosterone, dehydroepiandrosterone sulphate (DHEAS), insulin-like growth factor (IGF-I), sex hormone-binding globulin (SHBG), calcium, 25(OH) vitamin D, albumin, and creatinine clearance. The correlation between FN BMD and RASM (that includes upper and lower limb muscle mass) was of slightly greater magnitude than that between FN BMD and the relative upper limb muscle mass and between FN BMD and the relative leg muscle mass ( r = 0.39; p ≤ 0.001 versus r = 0.36; p ≤ 0.001 and r = 0.34; p ≤ 0.001, respectively). The stepwise multiple linear regression model showed that FN BMD was significantly associated with RASM (15% of FN BMD variance, p < 0.0001), age (10% of FN BMD variance, p < 0.0001), physical activities from age 11–20 years (5% of FN BMD variance, p < 0.01), and blood PTH, IGF-I, and creatinine clearance, (2%, 2%, and 1% of FN BMD variance, respectively, p < 0.05). These results show that RASM, with ASM measured by DXA, is the strongest factor associated with FN BMD in men. It remains to be determined whether assessing RASM by anthropometric methods would help screening adult men at risk of low FN BMD. Furthermore, since RASM is associated with FN BMD independently of appendicular skeletal loads and other lifestyle, nutritional, and hormonal factors, this suggests that common factors, possibly genetic factors, might also influence the coupled maintenance of appendicular muscle mass and FN BMD in adult men.