The transport property of natural fissured reservoir is highly related to the fracture permeability. The composite topography of natural fractures is always rough, with self-affine surfaces and non-uniform aperture field. All these would affect the transport properties significantly. Therefore, it is of fundamental importance to clarify the control mechanism on fluid flow through rough fractures with self-affine and mismatched surfaces. In this work, four controlling factors, namely local rough geometry, hydraulic tortuosity, surface tortuosity, and non-uniform aperture distribution were identified for the mismatched and rough fracture flows. Afterwards, the implications of these factors on fluid flow were specified by analytical derivation. Combined with the cubic law and the triple-effect model, a quadruple-effect permeability model was established to characterize the comprehensive effects from the composite topography of rough fractures. Moreover, the semi-empirical coefficient and index in surface roughness factor were determined by Lattice Boltzmann simulations, and provide fundamental support to define the effective hydraulic aperture which is a key parameter in the quadruple-effect model. Theoretical analysis denoted that the quadruple-effect model could generalize other models in the literature, and its performance was approved by numerical validation.