Predicting magnetic ordering in kagome compounds offers the possibility of harnessing topological or flat-band physical properties through tuning of the magnetism. Here, we examine the magnetic interactions and phases of ErMn6Sn6 which belongs to a family of RMn6Sn6, R = Sc, Y, Gd–Lu, compounds with magnetic kagome Mn layers, triangular R layers, and signatures of topological properties. Using results from single-crystal neutron diffraction and mean-field analysis, we find that ErMn6Sn6 sits close to the critical boundary separating the spiral-magnetic and ferrimagnetic ordered states typical for non-magnetic versus magnetic R layers, respectively. Finding interlayer magnetic interactions and easy-plane Mn magnetic anisotropy consistent with other members of the family, we predict the existence of a number of temperature and field dependent collinear, noncollinear, and noncoplanar magnetic phases. We show that thermal fluctuations of the Er magnetic moment, which act to weaken the Mn-Er interlayer magnetic interaction and quench the Er magnetic anisotropy, dictate magnetic phase stability. Our results provide a starting point and outline a multitude of possibilities for studying the behavior of Dirac fermions in RMn6Sn6 compounds with control of the Mn spin orientation and real-space spin chirality.