We investigated the physiological (heart rate variability) and psychological (state of anxiety, pleasantness, and comfort) effects of ambient bergamot scent on the stress levels of office workers by exposing them to the scent while stressors persisted as the workers continued to work on the office tasks. Forty-eight young adults were randomly assigned to either a control or scent group. Our results show that the stress restoration effect of bergamot scent depends on gender. The change in heart rate variability revealed that bergamot scent increased stress among males but not for females. The reported pleasantness and comfort followed the same trend. Compared to the control groups, females in the scent group thought the office smelled pleasant and felt more comfortable, but males in the scent group reported the opposite. However, no gender effect was found in the level of state anxiety. Specifically, compared to the control groups, both males and females exposed to the bergamot scent self-reported decreasing stress levels. This inconsistency between self-reported stress and physiological measurements is not uncommon, especially among males who are socialized to downplay emotional experiences. Our data suggest that there is indeed a gender difference in the effectiveness of the bergamot scent for reducing stress in office workers.