Based on the continuity of length scale for asperities, a complete model of normal stiffness between curved fractal surfaces considering friction factor is proposed. Through the coupling of critical base diameter and contact area, deformation modes of all asperities are determined, and contact stiffness of the whole rough surface is derived by double integral. Accordingly, effects of contact load, surface topography, friction factor, size parameters and material properties on the contact stiffness are investigated. Results show that contact stiffness depends sensitively on the fractal dimension, and exhibits nonmonotonic characteristics as fractal roughness varies across scales. Finally, predicted values of normal stiffness are compared with existing contact models as well as experimental data.