(1) Background: Atherosclerosis is a leading cause of vascular death worldwide. High urinary phosphate has recently been identified as a cardiovascular risk factor, but its role has not been fully established. The aim of this study was to investigate the association between urinary phosphate and subclinical atherosclerosis in the carotid, femoral as well as coronary territories; (2) Methods: We performed a cross-sectional analysis of a sample of 1169 middle-aged men, aged 50.9 years (SD 3.7), without previous cardiovascular disease, belonging to the Aragon Workers Health Study (AWHS). Urinary phosphate was analyzed in urine samples using the Fiske-Subbarow method. The presence of carotid plaque and femoral plaque was assessed by ultrasound and coronary artery calcium score (CACS) by computed tomography. Demographic, anthropometric and clinical data were collected at annual medical examinations. Logistic regression models were used to estimate the prevalence of adjusted atherosclerosis in the different vascular arteries; (3) Results: A significant inverse association was observed between urinary phosphate and subclinical atherosclerosis in the carotid [OR 95% CI 0.69 (0.49-0.99)] and coronary (CACS > 200) [OR 95% CI 0.46 (0.23-0.88)] arteries; however, no statistically significant association was found between urinary phosphate and the presence of atheroma plaques in the femoral territory [OR 1.02 (0.72-1.45)]; (4) Conclusions: In middle-aged men, a higher urinary phosphate concentration is associated with a lower prevalence of subclinical carotid and coronary atherosclerosis compared with those with a lower urinary phosphate concentration.