The exploration and exploitation of shale oil resources enhance the necessity to characterize the mobility of the occluded oil in shales. Up to now, the understanding of the controlling factors of shale oil mobility has mainly come from molecular dynamics simulation but lacking in-depth research on actual shales. Therefore, in this study, the effects of mineral composition, soluble organic matter (SOM) composition and wettability on the shale oil mobility were studied by X-ray diffraction, N2 adsorption, mercury intrusion, contact angle, chloroform bitumen and group component detection methods, comprehensively. The results show that: (1) The asphaltene contents and (Asphaltenes × Aromatics)/(Saturates × Resins) negatively and positively correlate with clay and carbonate mineral contents, respectively, while the I/S and illite contents present opposite trend with (Asphaltenes × Aromatics)/(Saturates × Resins). These correlations indicate that the SOM that confined in the pores constructed with carbonate minerals and illite present better flow potential. (2) Both δVt and δSSA negatively correlate with asphaltenes and (Asphaltenes × Aromatics)/(Saturates × Resins). That is, less pores and surfaces can be released by extraction for the shales with more asphaltenes. Therefore, it can be concluded that asphaltene is a negative factor for shale oil seepage. Additionally, the larger pores present better flow condition for shale oil, even for asphaltenes. (3) Pore surface wettability also significantly affects the shale oil mobility. The hydrophobicity of the pore wall greatly inhibits the shale oil flow, especially for the mesopores and macropores. The pattern diagram was established to illustrate the effect of the interactions between pore surfaces and shale oil components on the movable potential. Based on our research, mineral composition, group composition and pore surface wettability are efficient parameters for the shale oil mobility evaluation based on the interfacial interaction theory, and the “sweet spots” for shale oil seepage of the Jiyang Depression are the shales with more illite, carbonate, saturates and relatively low lipophilicity.