Shrubs and herbs are two important types of vegetation that play a pivotal role in combating land desertification in arid sandy land. Yet, the effects of plant litter on sandy soil microbial activities and soil nutrients during the shrub and herb colonization process are still unclear. Here, contributions of plant litter to soil microbial activity improvement and soil nutrient enhancement along herb and shrub colonization expansion interfaces were studied in an arid sandy landscape. Plant litter mass, soil water content, pH, nutrient content, extracellular enzyme activities, and microbial community structure (taxon composition, α-diversity, β-diversity, and co-occurrence pattern) of soils at depths of 0–10 cm were examined relative to the colonization processes of shrubs and herbs (shrub and herb vegetation → sand-shrub and sand-herb interface → a sand dune) in an arid sandy land. We found that the processes of both herb and shrub colonization significantly increased litter cumulative mass at the soil surface to soil depths of 5 cm, and shrub colonization contributed several folds more litter than herb colonization. Plant litter input significantly decreased soil pH, accelerated soil extracellular enzyme activities, increased abundances of some microbial phyla (Actinobacteria, Chloroflexi, Acidobacteria, Gemmatimonadetes, Patescibacteria, Armatimonadetes, Ascomycota, Moritierellomycota, and Chytridiomycota), enhanced both prokaryotic and fungal α-diversities, altered both prokaryotic and fungal communities, and promoted complexity of microbial networks during both herb and shrub colonization. As a result, microorganisms actively participated in litter decomposition and soil nutrient enhancement. Nevertheless, shrub colonization process exhibited greater potential in nutrient input than herb colonization process due to its greater plant litter input, though both colonization played a comparable role in accelerating soil enzyme activity and improving soil microbial community diversity and co-occurrence. Therefore, xeromorphic shrubs are still more important plants than herbs should be used for vegetation construction and ecological restoration in arid sandy land.