Background: Life expectancy has increased dramatically worldwide, resulting in a sharp rise of age-associated diseases, including late-onset hypogonadism (LOH) in men. Aim: To assess the prevalence of LOH and to determine its relationships with cardiovascular risk factors in healthy middle-aged men. Material and Methods: A total of 200 men aged 44–55 (median 48.44, interquartile range 45.02–52.50) undergoing regular medical examinations were enrolled in a study. All participants were asked to complete the International Index of Erectile Dysfunction, Aging Male's Symptoms, and Beck Depression Inventory questionnaires. Sex and gonadotropin hormones were measured in all participants. Cardiovascular diseases (CVD) risk was assessed in all participants by using the Systematic Coronary Risk Evaluation scale, and additional risk factors were recommended in the 2019 European Society of Cardiology and European Atherosclerosis Society guidelines for the management of dyslipidemias: lipid modification to reduce cardiovascular risk. Results: Ninety-eight (49%) men suffered from LOH. Men with total testosterone (TT) of <12 nmol/L were considered testosterone-deficient. The mean TT level was 8.9 ± 2.4 nmol/L in the LOH group. Clinical signs and symptoms of LOH were associated with lower TT levels: 7.78 ± 2.49 and 10.15 ± 1.32 in the groups with and without LOH, respectively (p = 0.004). The prevalence of obesity, metabolic syndrome, and depression was significantly higher among men with LOH, compared with men without it. High cardiovascular risk was found only in those participants present with LOH (p = 0.003). After collecting other risk factors, we found that 94.9% of participants assigned to the LOH group had high cardiovascular risk. Conclusion: The prevalence of LOH is high among healthy middle-aged men. Moreover, it is associated with an increased cardiovascular risk. Early measurement of TT levels in this population group may predict the development of CVD and allow the introduction of timely primary prevention measures to reduce cardiovascular morbidity and mortality.