Background: Semen quality is frequently used as a marker for male reproductive health, and psychological stress has been shown to affect the body systemically, causing disruptions in the autonomic and neuroendocrine systems, among others. Objective: Examine associations between self-reported daily stress and semen quality parameters among men trying to conceive. Methods: A prospective, preconception cohort of 183 heterosexual couples (women 18–35, men 18–40), without known infertility or subfertility, were recruited. Women observed cervical mucus to identify an estimated day of ovulation and fertile window (FW). Men recorded self-perceived stress daily on a scale from 0 (none) to 10 (high) during the woman’s FW. Semen samples were collected using a semen collection device after the FW and were frozen in the home freezer. 85 men having at least one semen sample and recorded stress measures were analyzed. Intrapersonal variability, which quantifies a man’s deviation from his own baseline stress, was calculated in addition to mean stress. Linear mixed-effects models for repeated measurements were used to estimate associations. Results: The geometric mean (GM) of the total sperm count was 161.4 (95% CI: 141.2, 184.5) M, and the GM for concentration was 59.6 (95% CI: 52.8, 67.4) M/mL. Models controlled for age, BMI, abstinence days prior to collection, mean stress over the previous 45 days, and interactions between age, BMI, and stress. For a unit increase in Z-score of a man’s variability in self-reported stress levels, while holding mean stress fixed at the population mean, his total sperm count decreased by 15.0% (95% CI: 3.1%, 25.4%), and his sperm concentration decreased by 14.6% (95% CI: 4.5%, 23.7%). No significant associations with stress and semen volume were observed. Discussion: High intrapersonal variability in a man’s stress level was associated with decreased total sperm count and decreased concentration, but not semen volume.