Heavy metal pollution seriously threatens the diversity and composition of microbial communities in various ecosystems. However, little is known about the effects of heavy metal pollution on the structure of microbial communities in the three habitats of "surface water-sediment-groundwater." Here, with help of 16S rRNA high-throughput sequencing technology, the diversity and composition of microbial communities, as well as the underlying controlling factors, were investigated and compared among the surface water, sediment, and groundwater of the Tanghe sewage reservoir. The results showed significant differences in the diversity of microbial communities among different habitats, with the highest α diversity observed in groundwater rather than surface water or sediment. Meanwhile, microbial communities also displayed distinct compositions among the three different habitats. Specifically, Pedobacter, Hydrogenophaga, Flavobacterium, and Algoriphagus were dominant in surface water; metal-tolerant bacteria such as Ornatilinea, Longilinea, Thermomarinilinea, and Bellilinea prevailedin sediment; and Arthrobacter, Gallionella, and Thiothrix were abundant in groundwater. Furthermore, physicochemical factors and metal variables jointly determined the microbial community structure in the three habitats. Among the influencing factors,pH, NO3--N, and Li were the main factors affecting the microbial structure in surface water; TP, NH4+-N, Cr, Fe, Cu, and Zn significantly impacted microorganisms in sediment; and only pH (not metal pollutants) was weakly associated with the microbial composition in groundwater. Overall, heavy metal pollution significantly shaped the microbial community structure in sediment, followed by that in surface water and groundwater. These results provide important scientific guidance for the ecological restoration and the sustainable development of heavy metal-polluted ecosystems.