Permeability hysteresis under cyclic pressure loading and unloading has received a lot of attention in both science and engineering. But most of the existing model is only for a one-time pressure cycle. Therefore, a permeability hysteresis model is established based on the theory of elastic-structural deformation of capillary cross section and the fractal theory of porous media. Both the triangular and quadrilateral structures are considered. The stress sensitivity of structural deformation decreases with the increase in the cycle. The porosity hysteresis can also be predicted by the proposed model. Compared with experimental data with different permeability hysteresis, the prediction of the proposed model is consistent with the experimental results. Compared with other models, the proposed model has a smaller average error and a better agreement with experimental data. The proposed model can predict the permeability hysteresis under not only a one-time pressure cycle like the existing model but also multiple pressure cycles. The influence of parameters shows that the decrease in Young’s modulus and Poisson’s ratio of the solid cluster increases the permeability stress sensitivity but does not influence the permeability hysteresis. The increase in the proportion of quadrilateral structure and stress sensitivity of structural deformation increases the permeability hysteresis and stress sensitivity at the same time, while the capillary fractal dimension, tortuosity fractal dimension, and the decay rate of stress sensitivity of structural deformation during the cycles show the opposite. The proposed model has a significant meaning in underground resource mining and the study of permeability hysteresis mechanism.