While the tendency to return to previously visited locations-termed 'site fidelity'-is common in animals, the cause of this behaviour is not well understood. One hypothesis is that site fidelity is shaped by an animal's environment, such that animals living in landscapes with predictable resources have stronger site fidelity. Site fidelity may also be conditional on the success of animals' recent visits to that location, and it may become stronger with age as the animal accumulates experience in their landscape. Finally, differences between species, such as the way memory shapes site attractiveness, may interact with environmental drivers to modulate the strength of site fidelity. We compared inter-year site fidelity in 669 individuals across eight ungulate species fitted with GPS collars and occupying a range of environmental conditions in North America and Africa. We used a distance-based index of site fidelity and tested hypothesized drivers of site fidelity using linear mixed effects models, while accounting for variation in annual range size. Mule deer Odocoileus hemionus and moose Alces alces exhibited relatively strong site fidelity, while wildebeest Connochaetes taurinus and barren-ground caribou Rangifer tarandus granti had relatively weak fidelity. Site fidelity was strongest in predictable landscapes where vegetative greening occurred at regular intervals over time (i.e. high temporal contingency). Species differed in their response to spatial heterogeneity in greenness (i.e. spatial constancy). Site fidelity varied seasonally in some species, but remained constant over time in others. Elk employed a 'win-stay, lose-switch' strategy, in which successful resource tracking in the springtime resulted in strong site fidelity the following spring. Site fidelity did not vary with age in any species tested. Our results provide support for the environmental hypothesis, particularly that regularity in vegetative phenology shapes the strength of site fidelity at the inter-annual scale. Large unexplained differences in site fidelity suggest that other factors, possibly species-specific differences in attraction to known sites, contribute to variation in the expression of this behaviour. Understanding drivers of variation in site fidelity across groups of organisms living in different environments provides important behavioural context for predicting how animals will respond to environmental change.