The diversity and distribution patterns of the abundant and rare microbial sub-communities in hot spring ecosystems and their assembly mechanisms are poorly understood. The present study investigated the diversity and distribution patterns of the total, abundant, conditionally rare, and always rare taxa in the low- and moderate-temperature hot spring sediments on the Tibetan Plateau based on high-throughput 16S rRNA gene sequencing, and explored their major environmental drivers. The diversity of these four bacterial taxa showed no significant change between the low-temperature and moderate-temperature hot spring sediments, whereas the bacterial compositions were obviously different. Stochasticity dominated the bacterial sub-community assemblages, while heterogeneous selection also played an important role in shaping the abundant and conditionally rare taxa between the low-temperature and moderate-temperature hot spring sediments. No significant difference in the topological properties of co-occurrence networks was found between the conditionally rare and abundant taxa, and the connections between the paired operational taxonomic units (OTUs) were almost positive. The diversity of the total, abundant, and conditionally rare taxa was governed by the salinity of hot spring sediments, while that of the always rare taxa was determined by the content of S element. In contrast, temperature had significant direct effect on the composition of the total, abundant, and conditionally rare taxa, but relatively weak influence on that of the always rare taxa. Besides, salinity was another major environmental factor driving the composition of the abundant and rare sub-communities in the hot spring sediments. These results reveal the assembly processes and major environmental drivers that shaped different bacterial sub-communities in the hot spring sediments on the Tibetan Plateau, and indicate the importance of conditionally rare taxa in constructing bacterial communities. These findings enhance the current understanding of the ecological mechanisms maintaining the ecosystem stability and services in extreme environment.