The impurity problems within vortex cores of two-dimensional s-wave and chiral p-wave superconductors are studied numerically in the framework of the quasiclassical theory of superconductivity and self-consistent Born approximation under a trial form of the pair potential. The dispersion and impurity scattering rate (the inverse of the relaxation time) of the Andreev bound state localized in vortex cores are deduced from the angular-resoloved local density of states. The energy dependence of the impurity scattering rates depends on the pairing symmetry; particularly, in the chiral p-wave vortex core where chirality and vorticity have opposite sign and hence the total angular momentum is zero, the impurities are ineffective and the scattering rate is vanishingly small. Owing to the cancellation of angular momentum between chirality and vorticity, the chiral p-wave vortex core is similar to locally realized s-wave region and therefore non-magnetic impurity is harmless as a consequence of Anderson's theorem. The results of the present study confirm the previous results of analytical study (J. Phys. Soc. Jpn. {\bf 69} (2000) 3378) in the Born limit.