Acorus tatarinowii, a famous traditional Chinese medicine, is used for the clinical treatment of memory impairment and dementia. In this research, AT50, the crude polysaccharide extracted from A. tatarinowii rhizome, significantly improved the memory and learning ability of mice with Alzheimer's disease (AD) and exerted excellent anti-neuroinflammatory effects. More importantly, AT50 returned the levels of NO, TNF-α, IL-1β, PGE-2, and IL-6 in AD mouse brains to normal levels. To identify the active ingredients in AT50, a heteropolysaccharide ATP50–3 was obtained from AT50. Structural analysis indicated ATP50–3 consisted of α-L-Araf-(1→, →2)-α-L-Araf-(1→, →3)-α-L-Araf-(1→, →5)-α-L-Araf-(1→, α-D-Xylp-(1→, →3,4)-β-D-Xylp-(1→, →3)-α-D-Galp-(1→, →3,6)-α-D-Galp-(1→, →6)-4-OAc-α-D-Galp-(1→, →3,4,6)-α-D-Galp-(1→, →4)-α-D-Glcp-(1→, →2,3,6)-β-D-Glcp-(1→, →4,6)-α-D-Manp-(1→, →3,4)-α-L-Rhap-(1→, →4)-α-D-GalpA-(1→, and →4)-α-D-GlcpA-(1 → residues and terminated with Xyl and Ara. Additionally, ATP50–3 significantly inhibited the release of proinflammatory factors in lipopolysaccharide-stimulated BV2 cells. ATP50–3 may be an active constituent of AT50, responsible for its anti-neuroinflammatory effects, with great potential to treat AD.