The low energy level structure of $N=20$ $^{32}\mathrm{Mg}$ obtained via \ensuremath{\beta}-delayed \ensuremath{\gamma} spectroscopy is reported. The level structure of $^{32}\mathrm{Mg}$ is found to be completely dominated by intruders. An inversion between the 1p-1h and 3p-3h states is observed for the negative parity states, similar to the 0p-0h and 2p-2h inversion for the positive parity states in these $N~20$ nuclei. The intruder excited states, both positive and negative parity, are reasonably explained by Monte Carlo shell model calculations, which suggest a shrinking $N=20$ shell gap with decreasing $Z$.