Few studies are available on the relationship of androstenedione with inflammation and obesity and the effect of androstenedione and inflammation on the association between testosterone and obesity. This study intended to examine the mediation effect of inflammatory markers on the association of testosterone with obesity and the moderation effect of androstenedione on the association of testosterone with inflammation and obesity in Chinese rural men. This cross-sectional research enrolled 2536 male rural inhabitants from the Henan Rural Cohort study. The serum concentrations of testosterone and androstenedione were determined by liquid chromatography-tandem mass spectrometry. Linear and logistic regression were used to examine the relationships between testosterone, inflammatory markers, and obesity. Mediation and moderation analyses were carried out to evaluate the potential effects of inflammatory markers on the relationship between testosterone and obesity, as well as androstenedione on the relationships of testosterone with inflammation and obesity. After adjusting for confounding factors, the results showed that testosterone and androstenedione were negatively related to obesity, and inflammatory markers were positively associated with obesity. Besides, testosterone and androstenedione were negatively associated with inflammatory markers. Mediation analysis showed that white blood cell, neutrophil, monocyte, and high-sensitivity C-reactive protein had mediating effects on the association between testosterone and obesity. The most vital mediator was high-sensitivity C-reactive protein, and its proportion of the effect was 11.02% (defined by waist circumference), 11.15% (defined by waist-to-hip ratio), 12.92% (defined by waist-to-height ratio), and full mediating effect (defined by body mass index). Moreover, androstenedione played negative moderation effects on the associations of testosterone with inflammation and obesity. Inflammatory markers and androstenedione were first found to have modifying effects on the association of testosterone with obesity. Higher levels of testosterone and androstenedione could reduce the inflammation level and risk of obesity, indicating their potential roles in the prevention and treatment of chronic diseases.