The acid resistance of cementitious materials is an important property when exposed to aggressive acidic environments. In this study, the waste glass powder (GP) and slag as precursors were activated by alkali (Na or K)-silicates or by a combination of (Na or K)-silicates with air pollution control (APC) residues to produce alkali-activated materials (AAMs) and alkali/sulfate-activated materials (ASAMs). The compressive strength and volume change of the AAMs and ASAMs mortar were performed to understand the influence of sulfuric acid attack on the resulting performances. Meanwhile, the microstructure and micro-mechanics evolution were investigated by the nanoindentation technology. The results showed that the compressive strengths of all AAMs and ASAMs mortar decreased under a low pH solution. The Na- and K-AAMs mortar exposed to the acid solution shrank considerably even though a large amount of gypsum formed, while the Na- and K-ASAMs exhibited certain expansion behavior due to the existing sodium sulfate. Upon the sulfuric acid attack, the framework changes of the K-A-S-H/N-A-S-K caused by dealumination had less effect on the integrity than the dealumination and decalcification of the C(-A)-S-H gel. In comparing the alkali cations, the K-type activator impacted more positively than the Na-type activator for the performance development of the AAMs and ASAMs under the sulfuric acid attack. Moreover, the AAMs and ASAMs showed a buffering capacity of acid. This work provides a scientific reference value for applying the AAMs and ASAMs in real environments.