Latent infection of the globally spread parasite Toxoplasma gondii in humans has been associated with changes in personality and behavior. Numerous studies have investigated the effect of toxoplasmosis on depression, but their results are inconsistent. Our study focused on the effect of latent toxoplasmosis on depression in men and women in association with their fertility. In 2016–2018, we recruited clients (677 men and 664 women) of the Center for Assisted Reproduction and asked them to complete a standardized Beck Depression Inventory-II. In women without fertility problems, we found higher depression scores in Toxoplasma-positive than in Toxoplasma-negative (p = 0.010, Cohen’s d = 0.48). Toxoplasma-positive infertile men, on the other hand, had lower depression scores than Toxoplasma-negative infertile men (p ≤ 0.001, Cohen’s d = 0.48). Our results are consistent with the previously described effects of latent toxoplasmosis, which seem to go in opposite directions regarding the effect on personality and behavior of men and women. Our results could be explained by gender-contrasting reactions to chronic stress associated with lifelong infection. This suggests that due to gender differences in the impact of latent toxoplasmosis, future studies ought to perform separate analyses for women and men.