Arsenic (As) and antimony (Sb) are both toxic metalloids that are of primary concern for human health. Mining activity has introduced elevated levels of arsenic and antimony into the rivers and has increased the risks of drinking water contamination in China. Due to their mobility, the majority of the metalloids originating from mining activities are deposited in the river sediments. Thus, depending on various geochemical conditions, sediment could either be a sink or source for As and Sb in the water column. Microbes are key mediators for biogeochemical transformation and can both mobilize or precipitate As and Sb. To further understand the microbial community responses to As and Sb contamination, sediment samples with different contamination levels were collected from three rivers. The result of the study suggested that the major portions of As and Sb were in strong association with the sediment matrix and considered nonbioavailable. These fractions, however, were also suggested to have profound influenceson the microbial community composition. As and Sb contamination caused strong reductions in microbial diversity in the heavily contaminated river sediments. Microorganisms were more sensitive to As comparing to Sb, as revealed by the co-occurrence network and random forest predictions. Operational taxonomic units (OTUs) that were potentially involved in As and Sb metabolism, such as Anaerolinea, Sphingomonas, and Opitutus, were enriched in the heavily contaminated samples. In contrast, many keystone taxa, including members of the Hyphomicrobiaceae and Bradyrhizobiaceae families, were inhibited by metalloid contamination, which could further impair crucial environmental services provided by these members.