Abstract We show that, under weak assumptions, the automorphism group of a $\textrm{CAT(0)}$ cube complex $X$ coincides with the automorphism group of Hagen’s contact graph $\mathcal{C}(X)$. The result holds, in particular, for universal covers of Salvetti complexes, where it provides an analogue of Ivanov’s theorem on curve graphs of non-sporadic surfaces. This highlights a contrast between contact graphs and Kim–Koberda extension graphs, which have much larger automorphism group. We also study contact graphs associated with Davis complexes of right-angled Coxeter groups. We show that these contact graphs are less well behaved and describe exactly when they have more automorphisms than the universal cover of the Davis complex.