The double selection of environment adaptation and host specificity forced the diversification of rhizobia in nature. In the tropical region of China, Medicago polymorpha and Medicago lupulina are widely distributed, particularly in purple soil. However, the local distribution and diversity of rhizobia associated with these legumes has not been systematically investigated. To this end, root nodules of M. polymorpha and M. lupulina grown in purple soil at seven locations in Yunnan Province of China were collected for rhizobial isolation. The obtained rhizobia were characterized by RFLP of 16S-23S rRNA intergenic spacer, BOXAIR fingerprinting, and phylogeny of housekeeping and symbiosis genes. As result, a total of 91 rhizobial strains were classified into species Sinorhizobium medicae and S. meliloti, while three nodC gene types were identified among them. S. medicae containing nodC of type I was dominant in farmlands associated with M. polymorpha; while S. meliloti harboring nodC of type III was dominant in wild land nodulated by M. lupulina. For both rhizobial species, greater genetic diversity was detected in the populations isolated from their preferred host plant. A high level of genetic differentiation was observed between the two Sinorhizobium species, and gene flow was evident within the populations of the same species derived from different soil types, indicating that rhizobial evolution is likely associated with the soil features. To examine the effects of environmental features on rhizobial distribution, soil physicochemical traits and rhizobial genotypes were applied for constrained analysis of principle coordinates, which demonstrated that soil features like pH, nitrogen and sodium were the principle factors governing the rhizobial geographical distribution. Altogether, both S. medicae and S. meliloti strains could naturally nodulate with M. polymorpha and M. lupulina, but the rhizobium-legume symbiosis compatibility determined by both the host species and soil factors was also highlighted.