It is generally accepted that sulfur can passivate the bioavailability of heavy metals in soil, but it is not clear whether high sulfur in cadmium (Cd) and chromium (Cr) contaminated soil has negative effect on soil microbial community and ecological function. In this study, total sulfur (TS) inhibited the Chao 1, Shannon, Phylogenetic diversity (Pd) of bacterial and Pd of fungi in slightly contaminated soil by Cd and Cr around pyrite. TS, total potassium, pH, total chromium, total cadmium, total nitrogen, soil organic matter were the predominant factors for soil microbial community; the contribution of TS in shaping bacterial and fungal communities ranked at first and fifth, respectively. Compared with the low sulfur group, the abundance of sulfur sensitive microorganisms Gemmatimonas, Pseudolabrys, MND1, and Schizothecium were decreased by 68.79–97.22% (p < 0.01) at high sulfur one; the carbon fixation, nitrogen cycling, phosphorus cycling and resistance genes abundance were significantly lower (p < 0.01) at the latter. Such variations were strongly and closely correlated to the suppression of energy metabolism (M00009, M00011, M00086) and carbon fixation (M00173, M00376) functional module genes abundance in the high sulfur group. Collectively, high sulfur significantly suppressed the abundances of functional microorganisms and functional genes in slightly contaminated soil with Cd and Cr, possibly through inhibition of energy metabolism and carbon fixation of functional microorganisms. This study provided new insights into the environmental behavior of sulfur in slightly contaminated soil with Cd and Cr.