Murrah and Nili-Ravi are water buffalo breeds widely used as dairy animals in Asian countries. In this study, we investigated the diversity of ruminal microbes in six Murrah and six Nili-Ravi water buffaloes consuming diets typical of those used in Southern China which had forage to concentrate ratios of 3.2 and 1.6. After feeding the diets for 25 days, ruminal fluid was sampled by stomach tube before the morning feeding. Bacterial and archaeal 16S rRNA genes and the ciliate protozoal 18S rRNA genes were PCR-amplified from DNA extracted from rumen samples, sequenced using 454 Titanium pyrosequencing, and analyzed using the QIIME software package. Our results showed that, at the phylum level, Bacteroidetes was the predominant bacterial group, accounting for 42–72% of total bacteria, followed by Firmicutes, Fibrobacteres, Proteobacteria and Spirochaetes. At genus level, Prevotella dominated, accounting for 22–58% of total bacteria, followed by Fibrobacter, Paludibacter, and Ruminococcus. While there were differences between the bacterial community compositions of different animals, there was no obvious correlation of bacterial community composition at the phylum or genus level with the diets or with buffalo breeds. Methanobrevibacter-related organisms were the dominant archaeal group, accounting for around 80% of the total, followed by Methanomassiliicoccales (Rumen Cluster C (RCC), 15%) and Methanosphaera (3%). Similar to the bacterial community, there was no obvious correlation of archaeal community profiles with diet or buffalo breed. The ciliate protozoal communities differed between the samples analyzed, and Entodinium was the most abundant group of ciliates in every sample, accounting for more than 40% of total protozoa. The second largest ciliate group varied in different samples, with Isotricha, Polyplastron or Dasytricha being the most dominant genera after Entodinium.