We demonstrate that a low-remanent sample magnetization ${M}_{\mathrm{R}}$ is crucial for all-optical magnetic switching (AOS) in ferrimagnets and ferrimagnet heterostructures. ${M}_{\mathrm{R}}$ may be devised by the composition of the material. However, it can also be controlled in situ by changing the sample temperature because it affects ${M}_{\mathrm{R}}$. We show that increasing the lattice temperature by laser pulses or a simple heating resistor enables AOS in a ferrimagnetic Tb-Fe film, which does not exhibit AOS at room temperature. We reconcile earlier contradicting results for AOS in heated and cooled magnetic films by applying the low-remanence criterion. It also applies to other existing rare-earth transition-metal (RE-TM) alloys, such as GdFeCo or Tb-Co, to ferrimagnet heterostructures, and to RE-free synthetic ferrimagnets.