In a context of long-term highly intensive grazing in grassland ecosystems, a better understanding of how quickly belowground biodiversity responds to grazing is required, especially for soil microbial diversity. In this study, we conducted a grazing experiment which included the CK (no grazing with a fenced enclosure undisturbed by livestock), light and heavy grazing treatments in a desert steppe in Inner Mongolia, China. Microbial diversity and soil chemical properties (i.e., pH value, organic carbon, inorganic nitrogen (IN, -N and -N), total carbon, nitrogen, phosphorus, and available phosphorus content) both in rhizosphere and non-rhizosphere soils were analyzed to explore the responses of microbial diversity to grazing intensity and the underlying mechanisms. The results showed that heavy grazing only deceased bacterial diversity in the non-rhizosphere soil, but had no any significant effects on fungal diversity regardless of rhizosphere or non-rhizosphere soils. Bacterial diversity in the rhizosphere soil was higher than that of non-rhizosphere soil only in the heavy grazing treatment. Also, heavy grazing significantly increased soil pH value but deceased NH4+-N and available phosphorus in the non-rhizosphere soil. Spearman correlation analysis showed that soil pH value was significantly negatively correlated with the bacterial diversity in the non-rhizosphere soil. Combined, our results suggest that heavy grazing decreased soil bacterial diversity in the non-rhizosphere soil by increasing soil pH value, which may be due to the accumulation of dung and urine from livestock. Our results highlight that soil pH value may be the main factor driving soil microbial diversity in grazing ecosystems, and these results can provide scientific basis for grassland management and ecological restoration in arid and semi-arid area.