It has been shown that a cosmological background with an anisotropic stress tensor, appropriate for a free streaming thermal neutrino background, can damp primordial gravitational waves after they enter the horizon, and can thus affect the CMB B-mode polarization signature due to such tensor modes. Here we generalize this result, and examine the sensitivity of this effect to non-zero neutrino masses, extra neutrino species, and also a possible relativistic background of axions from axion strings. In particular, additional neutrinos with cosmologically interesting neutrino masses at the O(1) eV level will noticeably reduce damping compared to massless neutrinos for gravitational wave modes with $k\tau_0 \approx 100-200$, where $\tau_0 \approx 2/H_0$ and $H_0$ is the present Hubble parameter, while an axion background would produce a phase-dependent damping distinct from that produced by neutrinos.