From adolescence, women become more likely to experience fear dysregulation. Oral contraceptives (OCs) can modulate the brain regions involved in fear processes. OCs are generally used for years and often initiated during adolescence, a sensitive period where certain brain regions involved in the fear circuitry are still undergoing important reorganization. It remains unknown whether OC use during adolescence may induce long-lasting changes in the fear circuitry. This study aimed to examine whether age of onset moderated the relationship between duration of use and fear-related brain structures. We collected structural MRI data in 98 healthy adult women (61 current users, 37 past users) and extracted grey matter volumes (GMV) and cortical thickness (CT) of key regions of the fear circuitry. Non-linear multiple regressions revealed interaction effects between age of onset and quadratic duration of use on GMV of the right hippocampus and right ventromedial prefrontal cortex (vmPFC). Among women who initiated OCs earlier in adolescence, a short duration of use was associated with smaller hippocampal GMV and thicker vmPFC compared to a longer duration of use. For both GMV and CT of the right vmPFC, women with an early OC onset had more grey matter at a short duration of use than those with a later onset. Our results suggest that OC use earlier in adolescence may induce lasting effects on structural correlates of fear learning and its regulation. These findings support further investigation into the timing of OC use to better comprehend how OCs could disrupt normal brain development processes.