Band topology is both constrained and enriched by the presence of symmetry. The importance of anti-unitary symmetries such as time reversal was recognized early on leading to the classification of topological band structures based on the ten-fold way. Since then, lattice point group and non-symmorphic symmetries have been seen to lead to a vast range of possible topologically nontrivial band structures many of which are realized in materials. In this paper we show that band topology is further enriched in many physically realizable instances where magnetic and lattice degrees of freedom are wholly or partially decoupled. The appropriate symmetry groups to describe general magnetic systems are the spin-space groups. Here we describe cases where spin-space groups are essential to understand the band topology in magnetic materials. We then focus on magnon band topology where the theory of spin-space groups has its simplest realization. We consider magnetic Hamiltonians with various types of coupling including Heisenberg and Kitaev couplings revealing a hierarchy of enhanced magnetic symmetry groups depending on the nature of the lattice and the couplings. We describe, in detail, the associated representation theory and compatibility relations thus characterizing symmetry-enforced constraints on the magnon bands revealing a proliferation of nodal points, lines, planes and volumes.