Organic farming is considered an effective form of sustainable agricultural management. However, understanding of soil microbial diversity and composition under long-term organic and conventional farming is still limited and controversial. In this study, the Illumina MiSeq platform was applied to investigate the responses of soil bacterial and fungal diversity and compositions to organic farming (OF) and improved conventional farming (CF, applied straw retention) in the rice-wheat rotation system. The results highlighted that the alpha diversity of microbial communities did not differ significantly, except for higher bacterial diversity under OF. However, there were significant differences in the compositions of the soil bacterial and fungal communities between organic and conventional farming. Under our experimental conditions, through the ecological functional analysis of significant different or unique bacterial and fungal taxonomic members at the phyla and genus level, OF enhanced nitrogen, sulfur, phosphorus and carbon dynamic cycling in soil with the presence of Nodosilinea, Nitrospira, LCP-6, HB118, Lyngbya, GOUTA19, Mesorhizobium, Sandaracinobacter, Syntrophobacter and Sphingosinicella, and has the potential to strengthen soil metabolic ability with Novosphingobium. On the other hand, CF increased the intensity of nitrogen cycling with Ardenscatena, KD1-23, Iamia, Nitrosovibrio and Devosia, but enriched several pathogen fungal members, including Coniochaeta, Corallomycetella, Cyclaneusma, Cystostereum, Fistulina, Curvularia and Dissoconium.