Suffusion, the process of particle migration and clogging within porous media, exerts a remarkable impact on various natural and engineered systems. Properties of the coarse-grained skeletons are crucial during suffusion, while the influence of the inherent anisotropy remains insufficiently explored. In this study, numerical model using the coupled computational fluid dynamics and the discrete element method (CFD-DEM) is performed to simulate the suffusion evolution of different fabric anisotropy configurations under controlled flow conditions. Granular skeletons are generated considering different initial rotational degrees of freedom (RDoFs) and orientations of irregular coarse particles, based on which the motion mechanism of migratory fines is investigated. The infiltration process is quantified by multi-scale parameters, including particle distribution, velocity evolution, visualization of pores and throats, force network, contact statistics and anisotropic characteristics. The results illustrate the non-negligible influence of inherent anisotropy on pore structure, consequently impacting the motion behavior of fine particles and the stress transmission within the granular skeletons. Such insights will contribute to advancing a fundamental understanding of suffusion, which enables the development of effective strategies for predicting and controlling in geotechnical and environmental engineering catastrophes.