We introduce a novel class of interaction-enabled topological crystalline insulators in two- and three-dimensional electronic systems, which we call ``topological crystalline magnet.'' It is protected by the product of the time-reversal symmetry $\mathcal{T}$ and a mirror symmetry or a rotation symmetry $\mathcal{R}$. A topological crystalline magnet exhibits two intriguing features: (i) it cannot be adiabatically connected to any Slater insulator and (ii) the edge state is robust against coupling electrons to the edge. These features are protected by the anomalous symmetry transformation property ${(\mathcal{R}\mathcal{T})}^{2}=\ensuremath{-}1$ of the edge state. An anisotropic response to the external magnetic field can be an experimental signature.