The anisotropy of the magnetic excitations in BaFe$_2$As$_2$ was studied by polarized inelastic neutron scattering which allows one to separate the components of the magnetic response. Despite the in-plane orientation of the static ordered moment we find the in-plane polarized magnons to exhibit a larger gap than the out-of-plane polarized ones indicating very strong single-ion anisotropy within the layers. It costs more energy to rotate a spin within the orthorhombic {\it a-b} plane than rotating it perpendicular to the FeAs layers.