Appropriate nitrogen (N) management is important to minimize N losses from intensively managed agricultural ecosystems. Understanding the community structure of arbuscular mycorrhizal fungi (AMF) in response to N management can be of great ecological significance, particularly with the recent emphasis on the role of AMF in N cycling. A comprehensive study of both the vertical distribution of AMF in the soil profile and the temporal changes in community structure in maize roots was conducted over a 5-year period at a field site on the North China Plain. The N treatments consisted of zero N, conventional farming practice, and optimum N based on an in-season soil Nmin test. Terminal restriction fragment length polymorphism and clone sequencing were used to analyse the AMF community. Optimum N mitigated the decline in richness of AMF in the conventional N treatment in the surface soil. Diverse and species-rich AMF communities occurred deep in the soil profile. A significant difference in AMF community structure was observed between the control and fertilizer N treatments but not between the two N application strategies. AMF communities deeper in the soil profile were subsets of those richer communities in the surface soil and the loss of AMF taxa was mostly due to the absence of rare taxa. Soil pH and Nmin contents were major soil properties affecting the soil AMF communities among the N treatments while vertical distribution was influenced mainly by soil electrical conductivity. Crop phenology had a stronger influence than N treatment on the temporal shifts in AMF communities in maize roots. Our results provide evidence for the importance of N management in maintaining AMF diversity. Changes in soil chemical properties due to N fertilization, in particular declining soil pH, should be integrated in N management strategies to reduce the negative impacts on AMF communities induced by N fertilization. Excessive N inputs induced significant changes in soil physicochemical properties, especially soil acidification, and may have negative impacts on AMF communities.