We numerically investigated how fine particle assemblages move through pore space of granular media filled with fluid on the assumption of extremely small Reynolds number and Stokes number. We calculated the particle trajectories passing through granular bed with uniform and nonuniform structures by the Stokesian dynamics method, which can take into account hydrodynamic interactions between particles. It was observed that the particle assemblage was moving complexly while avoiding granular bed, resulting in hydrodynamic diffusion. The hydrodynamic diffusive behavior in the traveling and lateral directions was evaluated using the index D h, which means the particles dispersion increases per unit time. By comparison with the uniform granular bed, it was found that the hydrodynamically-diffusive behavior in the nonuniform bed was quite distinctive. Particularly, in the lateral direction, the particle assemblages showed both positive and negative spreading depending on the non-uniformity. The present results indicate that the relationship between a nonuniform structure and the index D h could be applied to various engineering processes such as separating and sorting fine particles.