Recently, the role of lifestyle factors in testicular function has developed into a growing area of interest. Based on cross-sectional data on 3283 Taiwanese men, we investigated whether interactive effects of unhealthy lifestyle behaviors were associated with testicular function. The men were recruited from a private screening institute between 2009 and 2015. Lifestyle behaviors (smoking, alcohol drinking, physical activity (PA), sleeping habits, and diet) were obtained by a validated self-reported questionnaire. The men provided a semen sample and had blood drawn for sex hormone measurement. Men who smoked and drank had higher testosterone (T) levels (β = 0.81, p < 0.001) than those who neither smoked nor drank. Men who smoked and had high Western dietary pattern scores had higher T levels—by 0.38 ng/mL (p = 0.03). Those who drank and did not get enough sleep or had high Western dietary pattern scores had elevated T levels—by 0.60 ng/mL (p = 0.005) or 0.45 ng/mL (p = 0.02), respectively. Light PA and insomnia were associated with decreased T levels—by 0.64 ng/mL (p < 0.001). Those who smoked and drank or had light PA or had high Western dietary pattern scores had lower normal sperm morphologies (NSMs)—by 2.08%, 1.77%, and 2.29%, respectively. Moreover, drinkers who had high Western dietary pattern scores had higher sperm concentrations—by 4.63 M/mL (p = 0.04). Awareness and recognition of the long-term impact of lifestyle behaviors and better lifestyle choices may help to optimize the chance of conception amongst couples.