Stress, and the chronic overactivation of major stress hormones, is associated with several neuropsychiatric disorders. However, clinical literature on the exact role of stress either as a causative, triggering, or modulatory factor to mental illness remains unclear. We suggest that the impact of stress on the brain and behavior is heavily dependent on the developmental timing at which the stress has occurred, and as such, this may contribute to the overall variability reported on the association of stress and mental illness. Here, animal models provide a way to comprehensively assess the temporal impact of stress on behavior in a controlled manner. This review particularly focuses on the long-term impact of stress on behavior in various rodent stress models at three major developmental time points: early life, adolescence, and adulthood. We characterize the various stressor paradigms into physical, social, and pharmacological, and discuss commonalities and differences observed across these various stress-inducing methods. In addition, we discuss here how sex can influence the impact of stress at various developmental time points. We conclude here that early postnatal life and adolescence represent particular periods of vulnerability, but that stress exposure during early life can sometimes lead to resilience, particularly to fear-potentiated memories. In the adult brain, while shorter periods of stress tended to enhance spatial memory, longer periods caused impairments. Overall, males tended to be more vulnerable to the long-term effects of early life and adolescent stress, albeit very few studies incorporate both sexes, and further well-powered sex comparisons are needed.