As a global phenomenon, farmland abandonment continues to challenge the sustainability of the agri-food supply and rural development. Investigating the heterogeneous effects of multilevel location on farmland abandonment is of great importance to understand the spatial disparity and the mechanism of farmland abandonment, which has significant policy implications for food security and rural revitalization. Taking Tai’an City as a case, this study aims to explore the impact of multilevel location on farmland abandonment at the village level and its spatial heterogeneity. The results show that (1) high accessibility to regional centers and roads, rather than remoteness, leads to a high rate of farmland abandonment; (2) the effect of location varies depending on the level of location. High-level regional centers (city centers and county centers) and roads (national and provincial highways) exert a stronger impact on farmland abandonment than low-level town centers and county highways; (3) the effect of location is topographically heterogeneous due to the influence of terrain on the marginalization of farmland. In the plains, except for county highways, the distance to different levels of regional centers and roads is significantly negatively correlated with farmland abandonment. However, in mountainous areas, only high-level regional centers have significant negative impacts.