A study of metakaolin and limestone powder on sulfate resistance of cement-based materials was carried out by compressive strength changes, sulfate attack experiment as well as the microstructure development using X-ray diffraction (XRD), thermogravimetric analysis (TG/DTG), and 27Al magic-angle spinning nuclear magnetic resonance (27Al MAS NMR) techniques. It was found that metakaolin with 10% was optimum for compressive strength increase of mortars. While a dosage of 20% is required for improving sulfate resistance through significant consumption of calcium hydroxide, induction of Al into C-(A)-S-H rather than ettringite and the increase of mean chain length (MCL) of C-S-H. Limestone powder displayed certain chemical reactivity to promote the formation of thermostable product monocarboaluminate for the suppression of sulfate attack at some extent, while hybrids of metakaolin and limestone powder showed a negative sulfate resistance owing to the insufficient dosages of metakaolin within 5–10%. This study provided a constructive perspective on the using of metakaolin and limestone powder to improve sulfate resistance of cement-based materials.