In the field of soil hydrology, pore structure is a key factor affecting soil permeability. This paper uses micro-CT scanning technology to visualize and quantitatively analyze the impact of different pore sizes on the permeability of granite residual soil through scanning, image processing, pore removal, and permeability simulation. The results show that granite residual soil samples have a well-developed connected pore structure, with connected pore diameters ranging from 25-400 μm, while isolated pores are mostly smaller than 50 μm. Medium-sized pores bridge small and large pores, forming an important component of pore connectivity. Pore size significantly affects permeability; the removal of medium-sized pores greatly reduces the connectivity, fractal dimension, and permeability of the remaining pores, indicating that pores in the 100–300 μm range contribute the most to permeability. In contrast, pores smaller than 50 μm and larger than 300 μm have a minimal impact on overall permeability. Effective porosity and fractal dimension are crucial parameters affecting permeability, with a significant linear correlation between the two. This study supply crucial data that delineates the relationship between the structure of soil pores and their permeability, which is invaluable for developing improvement strategies aimed at reducing soil permeability.