Effects of spontaneous toroidal ordering on magnetic excitation are theoretically investigated for a localized spin model that includes a staggered Dzyaloshinsky-Moriya interaction and anisotropic exchange interactions, which arise from the antisymmetric spin-orbit coupling and the multi-orbital correlation effect. We show that the model exhibits a Neel-type antiferromagnetic order, which simultaneously accompanies a ferroic toroidal order. We find that the occurrence of toroidal order modulates the magnon dispersion in an asymmetric way with respect to the wave number: a toroidal dipole order on the zigzag chain leads to a band-bottom shift, while a toroidal octupole order on the honeycomb lattice gives rise to a valley splitting. These asymmetric magnon excitations could be a source of unusual magnetic responses, such as nonreciprocal magnon transport. A variety of modulations are discussed while changing the lattice and magnetic symmetries. Implications of candidate materials for asymmetric magnon excitations are presented.