Obesity and low serum testosterone (T) levels are interrelated and strongly influenced by dietary factors, and their alteration entails a great risk of hypogonadism. Substantial evidence suggests a bidirectional relationship between nutrient metabolism (e.g., glucose, lipids, and iron) and T levels in men; however, T-related dietary patterns remain unclear. This study investigated the dietary patterns associated with serum total T levels and its predictive effect on hypogonadism and the body composition. Anthropometry, blood biochemistry, and food frequency questionnaires were collected for 125 adult men. Dietary patterns were derived using a reduced rank regression from 32 food groups. Overall prevalence rates of central obesity and hypogonadism were 48.0% and 15.7%, respectively. An adjusted linear regression showed that age, insulin, red blood cell (RBC) aggregation, and transferrin saturation independently predicted serum total T levels (all p < 0.01). The total T-related dietary pattern (a high consumption of bread and pastries, dairy products, and desserts, eating out, and a low intake of homemade foods, noodles, and dark green vegetables) independently predicted hypogonadism (odds ratio: 5.72; 95% confidence interval: 1.11‒29.51, p < 0.05) for those with the highest dietary pattern scores (Q4) compared to those with the lowest (Q1). Scores were also negatively correlated with the skeletal muscle mass (p for trend = 0.002) but positively correlated with the total body fat mass (p for trend = 0.002), visceral fat mass (p for trend = 0.001), and to a lesser extent, subcutaneous fat mass (p for trend = 0.035) after adjusting for age. Randomized controlled trials are needed to confirm that improvement in dietary pattern can improve T levels and reduce hypogonadism.